Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.
Figures & tables
Figure 1: Spatial imagination is more than generating future RGB views. The same imagined state may support different spatial questions (e.g., top-down geometry helps global distance comparisons; panoramic views facilitate reasoning about open space), yet existing approaches typically expose it to the reasoner only through RGB. ImagiRep provides controlled access to the same underlying state through multiple representations, allowing us to study how the choice of representation affects downstream spatial reasoning.
Figure 2: Overview of ImagiRep . The top left illustrates benchmark construction, including a 3D scene, movement instructions, and observed view. The top right shows oracle RGB and six alternative representations of the same imagined state. The lower panels present example questions and ground-truth answers for the Viewpoint Imagination and Cross-view Integration question types.
Endpoint RGB + Additional Representation
Oracle
Task
Question Type
Observed View
+ Endpoint RGB
Top-down
Text
Video
Depth
Normals
Panorama
Selection
Viewpoint Imagination
Open-space Direction
60.4
57.1
74.7
69.2
65.9
64.8
61.5
73.6
89.0
Nearest-obstacle Direction
38.2
49.1
61.8
58.2
60.0
63.6
58.2
65.5
81.8
Visibility
39.4
64.4
73.1
71.2
76.0
73.1
75.0
76.9
80.8
Nearest Object
51.0
64.4
76.9
73.1
76.0
76.9
71.2
80.8
87.5
Farthest Object
68.4
58.2
75.5
67.3
72.4
70.4
67.3
71.4
87.8
Table 1: Complementary benefits of oracle representations . Oracle selection counts a question as correct if the endpoint RGB alone or any one of the additional representations answers it correctly. Videos are provided as 16 uniformly sampled frames. Best and second-best results are marked in bold and underlined, respectively.
Gemini 3.7 Flash
GPT-6.1-sol
Input
Viewpoint Imagination
Cross-view Integration
Viewpoint Imagination
Cross-view Integration
Random Selection
25.0
25.0
25.0
25.0
Question Only
31.7
29.7
31.0
31.0
Observed View(s)
53.7
66.3
57.2
68.0
+ Oracle Endpoint RGB
63.7
78.7
66.1
75.7
Table 2: Accuracy (%) of Gemini 3.7 Flash and GPT-6.1-sol under different input conditions.
+ Estimated Representation
Per-Type
Task
Question Type
Observed View
+ Est. Endpoint RGB
Top-down
Text
Video
Depth
Normals
Panorama
Best
Viewpoint Imagination
Open-space Direction
60.4
52.7
63.7
60.4
62.6
57.1
57.1
70.3
70.3
Nearest-obstacle Direction
38.2
45.5
56.4
54.5
61.8
54.5
52.7
60.0
61.8
Visibility
39.4
47.1
50.0
51.9
54.8
52.9
52.9
56.7
56.7
Nearest Object
51.0
49.0
57.7
56.7
59.6
61.5
58.7
59.6
61.5
Farthest Object
68.4
53.1
62.2
69.4
70.4
66.3
67.3
63.3
70.4
Table 3: Complementary benefits of estimated representations on Viewpoint Imagination. Reported scores retain correct predictions from estimated endpoint RGB and include additional questions answered correctly with each representation. Per-Type Best reports the highest score across representations for each question type. Best and second-best results among the six representations are marked in bold and underlined, respectively.
Question Type
Observed View
+ Est. RGB
+ Est. Depth
Nearest Object
56.8
74.1
79.0
Farthest Object
59.8
54.9
65.9
Spatial Ordering
79.6
89.8
93.4
All
68.0
76.0
82.0
Table 4: Complementary benefits of estimated representations on Cross-view Integration. Best and second-best results are marked in bold and underlined, respectively.
Figure 3: Example comparison of oracle and estimated spatial representations for Viewpoint Imagination . Oracle representations are from ground-truth 3D scenes; estimated representations are obtained from HY-World 2.0.
Figure 4: Example comparison of oracle and estimated representations for Cross-view Integration . Estimated RGB is synthesized by SEVA, and estimated depth is predicted from the synthesized RGB.
Figure 5: Overview of GEARS , combining prediction-agreement gating with adaptive representation selection.
Question Type
MindJourney
AVIC
GEARS
Viewpoint Imagination
Open-space Direction
53.8
51.6
63.7
Nearest-obstacle Direction
52.7
49.1
54.5
Visibility
45.2
46.2
48.1
Nearest Object
53.8
51.0
55.8
Farthest Object
69.4
63.3
68.4
Table 5: Comparison with test-time spatial imagination methods on ImagiRep . Accuracy (%) is reported for each question type, with the best results in bold.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Annotation interface for Viewpoint Imagination.
Task
Question Type
n
Viewpoint Imagination
Open-space Direction
91
Nearest-obstacle Direction
55
Visibility
104
Nearest Object
104
Farthest Object
98
Object-relative Direction
170
Appendix
Table 6: Number of questions per question type.
Endpoint RGB + Additional Representation
Task
Question Type
Observed View
+ Endpoint RGB
Top-down
Text
Video
Depth
Normals
Panorama
Oracle Selection
Viewpoint Imagination
Open-space Direction
52.7
58.2
67.0
63.7
63.7
69.2
63.7
74.7
85.7
Nearest-obstacle Direction
36.4
45.5
61.8
54.5
56.4
52.7
52.7
56.4
74.5
Visibility
32.7
49.0
57.7
57.7
55.8
56.7
51.9
55.8
69.2
Nearest Object
51.9
65.4
75.0
70.2
68.3
71.2
69.2
70.2
80.8
Farthest Object
66.3
65.3
75.5
68.4
69.4
73.5
69.4
71.4
80.6
Appendix
Table 7: Results with Gemini 3.7 Flash using oracle representations, under the protocol of Table 1 . Oracle selection: correct if the endpoint RGB or any additional representation is correct (an upper bound). Bold and underline: best and second-best additional representation.
Question Type
Added Representation
Endpoint Correct
Rescued
Total
Acc. (%)
Gain (pp)
Open-space Direction
Panorama
53
15
68/91
74.7
16.5
Nearest-obstacle Direction
Top-down
25
9
34/55
61.8
16.4
Visibility
Text
51
9
60/104
57.7
8.7
Nearest Object
Top-down
68
10
78/104
75.0
9.6
Farthest Object
Top-down
64
10
74/98
75.5
10.2
Object-relative Direction
Video
135
14
149/170
87.6
8.2
Appendix
Table 8: Adding the best representation per Viewpoint Imagination question type. Rescued: endpoint-RGB errors corrected by the added representation. Gain: rescued questions as a percentage of the type.
Viewpoint Imagination ( n=622 )
Cross-view Integration ( n=300 )
Representation
Acc.
Δ
Best For
Acc.
Δ
Best For
Endpoint RGB Only
63.7
–
–
78.7
–
–
+ Top-down
73.0
↑ 9.3
4/6
85.0
↑ 6.3
1/3
+ Text
69.3
↑ 5.6
1/6
83.3
↑ 4.7
0/3
+ Video
69.9
↑ 6.3
1/6
85.3
↑ 6.7
1/3
+ Depth
70.7
↑ 7.1
0/6
84.0
↑ 5.3
1/3
Appendix
Table 9: Representation breakdown for Gemini 3.7 Flash. Δ : change from the endpoint RGB alone. Best For: number of question types on which the representation scores highest (ties counted for each). Oracle selection as in Table 7 .
Figure 7: Examples of oracle and estimated spatial representations. Oracle representations are derived from ground-truth 3D scenes; estimated representations are obtained from HY-World 2.0 for Viewpoint Imagination.
Task
No Gate
+Gating
+Routing
Gating+ Routing
Viewpoint Imagination
54.7
58.8
58.5
61.4
Cross-view Integration
77.3
76.3
77.7
77.7
Appendix
Table 10: Gating ablation with GPT-6.1-sol on ImagiRep : overall accuracy (%). Best per row in bold.
Figure 8: Routing prompt of GEARS for Viewpoint Imagination. The output is constrained to a JSON object with a reason and one choice.
Figure 9: Answering prompt of GEARS . The gating stage queries it without and with the estimated endpoint RGB; the final answer uses the evidence selected by routing.
Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.
Mahtab Bigverdi, Linjie Li, Weikai Huang +8
University of Washington · 2Allen Institute for AI · 3Microsoft +1
Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.
Spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, remains a core challenge for multimodal large language models. Despite promising performance, recent multimodal large language models (MLLMs) often exhibit fragile reasoning traces in spatial intelligence tasks that involve consistent spatial state recognition. We argue that these failures stem from a mismatch between the spatial recognition mechanism and the text-only reasoning behavior of these MLLMs. Effective spatial reasoning requires low-level geometric structure to be faithfully preserved and updated throughout the reasoning process, whereas textual representations tend to abstract away precisely these critical details. To address this issue, we propose SpatialImaginer, a unified multimodal generation framework that integrates textual reasoning with visual imagination. Our framework adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation. To support this capability, we further introduce a difficulty-aware data engine with closed-loop verification to train the model to invoke visual imagination selectively when stable spatial state tracking is required. Extensive experiments on diverse spatial intelligence benchmarks show that SpatialImaginer achieves state-of-the-art performance and substantially improves robustness on complex multi-step spatial reasoning tasks.
Yian Li, Yang Jiao, Bin Zhu +4
College of Computer Science and Artificial Intelligence, Fudan University · School of Computing and Information Systems, Singapore Management University · School of Computer Science and Technology, East China Normal University +2