Organizations: 1*East China Normal University, China. · 3Shanghai Innovation Institute, China. · 2Shanghai Artificial Intelligence Laboratory, China. · University of Agder, Norway.
Spatial intelligence is central to embodied cognition, yet contemporary AI systems still struggle to reason about physical interactions in open-world human environments. Despite strong performance on controlled benchmarks, vision-language models often fail to jointly model physical dynamics, reference frames, and the latent human intentions that drive spatial change. We introduce Teleo-Spatial Intelligence (TSI), which captures an important aspect of spatial reasoning: integrating physical dynamics and action goals to reason about actions and their spatial consequences. To evaluate TSI, we present EscherVerse, a large-scale open-world resource built from 11,328 real-world videos, including an 8,000-example benchmark and a 35,963-example instruction-tuning set. Across 27 state-of-the-art vision-language models and an independent analysis of first-pass human responses from 11 annotators, we identify a persistent teleo-spatial reasoning gap: the strongest proprietary model achieves 57.26% overall accuracy, far below first-pass human performance, which ranges from 84.81% to 95.14% with a mean of 90.62%. Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it. EscherVerse provides a diagnostic testbed for purpose-aware spatial reasoning and highlights a critical gap between pattern recognition and human-level understanding in embodied AI.
Figures & tables
Figure 1: Teleo-Spatial Intelligence (TSI) contrasted with the current spatial understanding paradigm. Left, conventional spatial understanding is primarily object-centred, geometry-focused, and grounded in current-state description. Right, TSI extends spatial reasoning toward human-centred, goal-directed, and intent-aware understanding of future spatial change. Example questions illustrate the difference between static geometric queries and intent-conditioned reasoning about dynamic outcomes.
Figure 2: Conceptual design and evaluation of EscherVerse. a, Comparison of geometry-centred perception with goal-directed spatial reasoning, with the six benchmark categories shown below. b, Schematic motivation for evaluating models on real-world video tasks. c, Category-level accuracy profiles for the eight models named in the key; each coloured line represents one model. TSI, Teleo-Spatial Intelligence. Source data are provided as a Source Data file.
Figure 3: EscherVerse examples and construction. a, Example questions from the six categories, presented as text case cards with model and human responses. b, Stages of video selection and question screening. c, Benchmark composition by category (left) and question format (right); coloured sectors identify the categories and formats listed in the keys. d, Question generation and verification pipeline, including model screening and human review; video symbols are schematic. Original video links and clip time ranges for panel a are listed in Supplementary Table 17. QA, question–answer; LLM, large language model; VLM, vision-language model. Source data are provided as a Source Data file.
Figure 4: Independent first-pass human performance exceeds current multimodal models on EscherVerse. Human performance is shown as the distribution of overall accuracies across 11 annotators, each of whom answered all 8,000 benchmark questions under the first-pass protocol. Current frontier proprietary and open-weight models remain substantially below the human range, with the best proprietary model trailing the human mean by 33.36 percentage points. Source data are provided as a Source Data file.
Figure 5: The impact of real-world spatial instruction tuning. a, Accuracy profiles across six spatial-reasoning categories for Escher-8B, the simulation-trained ISI-8B baseline, Qwen3-VL-8B and Gemini-2.5-pro; the differences vary by category. b, Overall accuracy plotted against model parameter count for open-weight models. Closed-source models are shown in a separate display category labelled Closed (Est.); their horizontal positions are illustrative and do not represent verified parameter counts. c, Overall accuracy on EscherVerse for ViCA2, the Qwen3 base model, simulation-trained ISI-8B and Escher-8B after real-world instruction tuning. Source data are provided as a Source Data file.
Figure 6: Benchmarking spatial intelligence across diverse modalities and reasoning types. a, Heatmap visualization of accuracy across six spatial-reasoning categories. Escher-8B (orange text) outperforms the simulation-trained ISI-8B (blue text) and reduces the observed gap to the closed-source models in the top rows. b, Category-wise comparison between ISI-8B and Escher-8B. The largest annotated gains are in Action & Intent and Dynamic Spatial Relationships, followed by Object Deformation, consistent with a benefit from the EscherVerse-35k training data on these benchmark categories. c, Performance by question format. Patterned bars (hatched for Escher and dotted for ISI) show the observed accuracies, including multiple-select questions. Source data are provided as a Source Data file.
Figure 7: Analysis of model behaviours and failure cases. a, Hierarchical clustering of Table 1 accuracies across six spatial-reasoning categories and two question formats (multiple-select and single-choice). Each column is standardized across the ten displayed models by subtracting its mean and dividing by its population standard deviation. Ward linkage with Euclidean distance groups the standardized profiles; positive Z-scores indicate above-average accuracy within this model subset. The two Escher-8B variants cluster together, then with Qwen3-VL-32B-Thinking, and subsequently with GPT-5 and Gemini-2.5-pro. These similarities do not establish a shared internal reasoning mechanism. b, The robustness-complexity landscape. The x-axis represents overall accuracy, while the y-axis denotes the Robustness Ratio (accuracy on multiple-select questions divided by accuracy on single-choice questions). Bubble area is proportional to accuracy on Object Deformation. Both panels use the reported aggregate accuracies; the asterisk denotes Escher-8B-Instruct, and Think denotes its Thinking variant. The background zones indicate higher and lower observed robustness ratios. The arrow shows the performance difference between ISI-8B and Escher-8B under these reported metrics. c, Tracking failure: the model does not correctly integrate simultaneous egocentric and allocentric motion. d, Perspective failure: baseline models misidentify the golf ball’s final position relative to the person specified in the question. e, Reference-frame failure: baseline models identify the wave or water rather than the shore when asked about the surfer’s orientation. Original video links and clip time ranges for panels c–e are listed in Supplementary Table 17. Source data are provided as a Source Data file.
Model
Spatial Reasoning Abilities
Overall
Centricity
Question Types
Obj. Perm. & Occl.
Dynamic Spatial
Action & Intent
Predictive & Count.
Obj. Def. & State
Ego. vs. Allo.
Overall Acc.
Human- Centric
Object- Centric
Fill-in Blank
Multi- Select
Single Choice
True/ False
Open-Source Models
Qwen3-VL-2B-Instruct
39.87
38.24
43.96
38.55
39.34
37.90
38.90
39.88
35.74
30.72
7.80
34.03
50.35
Qwen2.5-VL-3B-Instruct
39.13
38.98
47.13
41.89
44.55
40.15
40.61
41.12
38.96
36.41
11.36
35.46
50.99
Qwen2.5-VL-7B-Instruct
43.37
37.31
46.83
40.03
39.34
34.90
38.66
38.49
39.22
27.96
13.36
33.07
50.74
Qwen3-VL-4B-Instruct
49.17
44.47
51.66
44.98
54.03
42.96
45.60
46.06
44.12
47.51
34.08
38.44
53.93
Table 1: Comprehensive evaluation of Vision-Language Models on the spatial reasoning benchmark. Accuracy (%) is reported for all models. Independent human performance is reported as overall first-pass accuracy only, because category-level human breakdowns are not part of this table. Source data are provided as a Source Data file.
Condition
Intent
Δ Base
Dynamic
Δ Base
Base
50.21%
–
37.65%
–
Intent-focused
57.51% (3.35)
+7.30
39.09% (0.00)
+1.44
Dynamic-only
54.53% (1.16)
+4.32
43.62% (1.16)
+5.97
Random mixture
52.16% (1.02)
+1.95
41.77% (1.45)
+4.12
Table 2: Equal-budget SFT ablation. Parentheses report the standard deviation across two training seeds rather than confidence intervals. Source data are provided as a Source Data file.
Primary label
n
Weighted
95% interval
Spatiotemporal grounding
86
28.8%
24.0–34.2
Perspective/reference frame
72
23.7%
19.3–28.9
Physical prediction
43
14.2%
10.7–18.6
Perception/recognition
36
12.1%
8.9–16.3
Benchmark ambiguity
29
9.6%
6.8–13.5
Action–goal binding
22
7.4%
5.0–11.0
Table 3: Adjudicated primary error distribution. Weighted intervals use a Kish effective sample size and are descriptive. Source data are provided as a Source Data file.
Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.
Spatial reasoning requires understanding how objects and observers are arranged in a scene and how their relations change with viewpoint or movement. Vision-language models (VLMs) typically perform both steps through model-internal reasoning, forming an implicit scene interpretation from partial views and imagining the consequences of spatial transformations. A misread scene can lead to a wrong answer, while even a correct interpretation does not ensure that imagined transformations satisfy spatial constraints. We introduce Spatial Harness, which represents scenes as explicit and executable world states to ground spatial reasoning in physics-engine feedback. A Perceiver integrates scene constraints extracted from the question with specialized visual estimates to place objects and cameras in a shared coordinate frame. The resulting state is instantiated in a physics engine, where a Reasoner executes viewpoint changes and uses computed measurements and rendered views to guide reasoning. We evaluate on MindCube, SAT, MMSI-Bench, and SpatialRGPT-Bench with open- and closed-weight VLMs. On MindCube, Spatial Harness improves Qwen3.5-27B from 52.70% to 71.14%, surpassing pySpatial by 7.52 percentage points. Given identical annotated object and camera poses, it reaches 98.25% versus 71.81% for model-internal reasoning. Code is publicly available at: https://github.com/tmlr-group/Spatial-Harness
Chentao Cao, Zhanke Zhou, Sangni Duan +2
TMLR Group, Department of Computer Science, Hong Kong Baptist University · ByteDance Seed
Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.