Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Zengqi Zhao; Zun Wang; Shoubin Yu; Mohit Bansal; Yue Zhang

cs.CVOct 27, 2026

Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Authors: Zengqi Zhao, Zun Wang, Shoubin Yu, Mohit Bansal, Yue Zhang

Organizations: UNC Chapel Hill

Abstract

Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.

Figures & tables

Figure 1: Spatial imagination is more than generating future RGB views. The same imagined state may support different spatial questions (e.g., top-down geometry helps global distance comparisons; panoramic views facilitate reasoning about open space), yet existing approaches typically expose it to the reasoner only through RGB. ImagiRep provides controlled access to the same underlying state through multiple representations, allowing us to study how the choice of representation affects downstream spatial reasoning.

Figure 2: Overview of ImagiRep . The top left illustrates benchmark construction, including a 3D scene, movement instructions, and observed view. The top right shows oracle RGB and six alternative representations of the same imagined state. The lower panels present example questions and ground-truth answers for the Viewpoint Imagination and Cross-view Integration question types.

				Endpoint RGB + Additional Representation						Oracle
Task	Question Type	Observed View	+ Endpoint RGB	Top-down	Text	Video	Depth	Normals	Panorama	Selection
Viewpoint Imagination	Open-space Direction	60.4	57.1	74.7	69.2	65.9	64.8	61.5	73.6	89.0
	Nearest-obstacle Direction	38.2	49.1	61.8	58.2	60.0	63.6	58.2	65.5	81.8
	Visibility	39.4	64.4	73.1	71.2	76.0	73.1	75.0	76.9	80.8
	Nearest Object	51.0	64.4	76.9	73.1	76.0	76.9	71.2	80.8	87.5
	Farthest Object	68.4	58.2	75.5	67.3	72.4	70.4	67.3	71.4	87.8

Table 1: Complementary benefits of oracle representations . Oracle selection counts a question as correct if the endpoint RGB alone or any one of the additional representations answers it correctly. Videos are provided as 16 uniformly sampled frames. Best and second-best results are marked in bold and underlined, respectively.

	Gemini 3.7 Flash		GPT-6.1-sol
Input	Viewpoint Imagination	Cross-view Integration	Viewpoint Imagination	Cross-view Integration
Random Selection	25.0	25.0	25.0	25.0
Question Only	31.7	29.7	31.0	31.0
Observed View(s)	53.7	66.3	57.2	68.0
+ Oracle Endpoint RGB	63.7	78.7	66.1	75.7

Table 2: Accuracy (%) of Gemini 3.7 Flash and GPT-6.1-sol under different input conditions.

				+ Estimated Representation						Per-Type
Task	Question Type	Observed View	+ Est. Endpoint RGB	Top-down	Text	Video	Depth	Normals	Panorama	Best
Viewpoint Imagination	Open-space Direction	60.4	52.7	63.7	60.4	62.6	57.1	57.1	70.3	70.3
	Nearest-obstacle Direction	38.2	45.5	56.4	54.5	61.8	54.5	52.7	60.0	61.8
	Visibility	39.4	47.1	50.0	51.9	54.8	52.9	52.9	56.7	56.7
	Nearest Object	51.0	49.0	57.7	56.7	59.6	61.5	58.7	59.6	61.5
	Farthest Object	68.4	53.1	62.2	69.4	70.4	66.3	67.3	63.3	70.4

Table 3: Complementary benefits of estimated representations on Viewpoint Imagination. Reported scores retain correct predictions from estimated endpoint RGB and include additional questions answered correctly with each representation. Per-Type Best reports the highest score across representations for each question type. Best and second-best results among the six representations are marked in bold and underlined, respectively.

Question Type	Observed View	+ Est. RGB	+ Est. Depth
Nearest Object	56.8	74.1	79.0
Farthest Object	59.8	54.9	65.9
Spatial Ordering	79.6	89.8	93.4
All	68.0	76.0	82.0

Table 4: Complementary benefits of estimated representations on Cross-view Integration. Best and second-best results are marked in bold and underlined, respectively.

Figure 3: Example comparison of oracle and estimated spatial representations for Viewpoint Imagination . Oracle representations are from ground-truth 3D scenes; estimated representations are obtained from HY-World 2.0.

Figure 4: Example comparison of oracle and estimated representations for Cross-view Integration . Estimated RGB is synthesized by SEVA, and estimated depth is predicted from the synthesized RGB.

Figure 5: Overview of GEARS , combining prediction-agreement gating with adaptive representation selection.

Question Type	MindJourney	AVIC	GEARS
Viewpoint Imagination
Open-space Direction	53.8	51.6	63.7
Nearest-obstacle Direction	52.7	49.1	54.5
Visibility	45.2	46.2	48.1
Nearest Object	53.8	51.0	55.8
Farthest Object	69.4	63.3	68.4

Table 5: Comparison with test-time spatial imagination methods on ImagiRep . Accuracy (%) is reported for each question type, with the best results in bold.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Annotation interface for Viewpoint Imagination.

Task	Question Type	$n$
Viewpoint Imagination	Open-space Direction	91
	Nearest-obstacle Direction	55
	Visibility	104
	Nearest Object	104
	Farthest Object	98
	Object-relative Direction	170

Appendix

Table 6: Number of questions per question type.

				Endpoint RGB + Additional Representation
Task	Question Type	Observed View	+ Endpoint RGB	Top-down	Text	Video	Depth	Normals	Panorama	Oracle Selection
Viewpoint Imagination	Open-space Direction	52.7	58.2	67.0	63.7	63.7	69.2	63.7	74.7	85.7
	Nearest-obstacle Direction	36.4	45.5	61.8	54.5	56.4	52.7	52.7	56.4	74.5
	Visibility	32.7	49.0	57.7	57.7	55.8	56.7	51.9	55.8	69.2
	Nearest Object	51.9	65.4	75.0	70.2	68.3	71.2	69.2	70.2	80.8
	Farthest Object	66.3	65.3	75.5	68.4	69.4	73.5	69.4	71.4	80.6

Appendix

Table 7: Results with Gemini 3.7 Flash using oracle representations, under the protocol of Table 1 . Oracle selection: correct if the endpoint RGB or any additional representation is correct (an upper bound). Bold and underline: best and second-best additional representation.

Question Type	Added Representation	Endpoint Correct	Rescued	Total	Acc. (%)	Gain (pp)
Open-space Direction	Panorama	53	15	68/91	74.7	16.5
Nearest-obstacle Direction	Top-down	25	9	34/55	61.8	16.4
Visibility	Text	51	9	60/104	57.7	8.7
Nearest Object	Top-down	68	10	78/104	75.0	9.6
Farthest Object	Top-down	64	10	74/98	75.5	10.2
Object-relative Direction	Video	135	14	149/170	87.6	8.2

Appendix

Table 8: Adding the best representation per Viewpoint Imagination question type. Rescued: endpoint-RGB errors corrected by the added representation. Gain: rescued questions as a percentage of the type.

	Viewpoint Imagination ( $n{=}622$ )			Cross-view Integration ( $n{=}300$ )
Representation	Acc.	$\Delta$	Best For	Acc.	$\Delta$	Best For
Endpoint RGB Only	63.7	–	–	78.7	–	–
+ Top-down	73.0	$\uparrow$ 9.3	4/6	85.0	$\uparrow$ 6.3	1/3
+ Text	69.3	$\uparrow$ 5.6	1/6	83.3	$\uparrow$ 4.7	0/3
+ Video	69.9	$\uparrow$ 6.3	1/6	85.3	$\uparrow$ 6.7	1/3
+ Depth	70.7	$\uparrow$ 7.1	0/6	84.0	$\uparrow$ 5.3	1/3

Appendix

Table 9: Representation breakdown for Gemini 3.7 Flash.

\Delta

: change from the endpoint RGB alone. Best For: number of question types on which the representation scores highest (ties counted for each). Oracle selection as in Table 7 .

Figure 7: Examples of oracle and estimated spatial representations. Oracle representations are derived from ground-truth 3D scenes; estimated representations are obtained from HY-World 2.0 for Viewpoint Imagination.

Task	No Gate	+Gating	+Routing	Gating+ Routing
Viewpoint Imagination	54.7	58.8	58.5	61.4
Cross-view Integration	77.3	76.3	77.7	77.7

Appendix

Table 10: Gating ablation with GPT-6.1-sol on ImagiRep : overall accuracy (%). Best per row in bold.

Panel of Figure 8 — Figure 8: Routing prompt of GEARS for Viewpoint Imagination. The output is constrained to a JSON object with a reason and one choice.

Panel of Figure 9 — Figure 9: Answering prompt of GEARS . The gating stage queries it without and with the estimated endpoint RGB; the final answer uses the evidence selected by routing.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Jun 2, 2026Mahtab Bigverdi, Linjie Li, Weikai Huang +8Multimodal Understanding Spatial Reasoning
University of Washington · 2Allen Institute for AI · 3Microsoft +1
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning
Apr 19, 2026Yian Li, Yang Jiao, Bin Zhu +4Multimodal CoT Reasoning Unified Multimodal Models
College of Computer Science and Artificial Intelligence, Fudan University · School of Computing and Information Systems, Singapore Management University · School of Computer Science and Technology, East China Normal University +2

Jun 2, 2026J/K move · Enter open · S save

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu+7

University of Washington · 2Allen Institute for AI · 3Microsoft+1

Multimodal Understanding Spatial Reasoning Visual Spatial Reasoning Visual Reasoning Latent Visual Reasoning VLM Reasoning+3

Open paper AskPDF arXiv

Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.

Abstract

Visual Spatial Reasoning

Explore similar work

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning