Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

Mimo Shirasaka; Haochen Zhang; Yonatan Bisk

cs.CVSep 7, 2026

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

Authors: Mimo Shirasaka, Haochen Zhang, Yonatan Bisk

Organizations: The University of Tokyo Tokyo, Japan · Carnegie Mellon University Pittsburgh, PA, USA

Abstract

Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization. Our project page is available at https://mimo-owl.github.io/POVBench/.

Figures & tables

Figure 1: Overview of contextual observer grounding. Models infer the observer’s location ( operational reasoning ), reconstruct their perspective ( observer grounding ), and localize the target ( relational reasoning ). Conditions A: Inferred , B: Stated , and C: Given require three, two, and one of these steps, respectively.

Figure 2: Overview of POVBench. Multi-view exploration images are collected along a simulated robot trajectory in ProcTHOR. For each observer–anchor pair, directional targets are defined around the anchor object from the observer’s viewpoint. Three grounding conditions ( A , B , C ) vary in how explicitly the sentence specifies the observer viewpoint.

Figure 3: Normalized L2 localization error by spatial direction on POVBench, for three representative models: Gemma-4, GPT-5.4, and Qwen3-VL-8B (Transformer).

Model	Inferred	Stated	Given
MolmoPoint-8B	0.294	0.282	0.237
GPT-5.4	0.315	0.311	0.189
Qwen3-VL-8B (Transformer)	0.372	0.369	0.296
Qwen3-VL-8B (vLLM)	0.359	0.356	0.291
Qwen3-VL-32B (vLLM)	0.449	0.440	0.488
InternVL3-38B	0.372	0.364	0.232

Table 1: Main benchmark results on POVBench. Mean normalized L2 localization errors (lower is better), averaged over three runs and spatial directions, computed over answered instances. Errors are generally lowest for Given and highest for Inferred ; the small Inferred – Stated gap indicates that explicitly naming the observer offers limited benefit. RoboPoint and Llama-3.2-Vision are evaluated only under the Given condition. The best value in each condition is in bold.

Intervention	Inferred	Stated	Resp.
Basic	0.372	0.369	100%
O-Plot	0.398	0.401	100%
O&A-Plot	0.342	0.343	100%
CoT	0.322	0.333	79%
S-CoT	0.294	0.302	85%

Table 2: Ablation results on POVBench (Qwen3-VL-8B (Transformer)). Mean normalized L2 errors, averaged across spatial directions and computed over answered instances under the same protocol as Table 1 . Resp. is the response rate.

Figure 4: Thirty exploration frames extracted from the first house-tour video Noori (2025) . These images serve as the visual context for the real-world evaluation.

Figure 5: Qualitative comparison between standard and Spatial-CoT ( S-CoT ) prompting on real-world observer-grounding scenarios. S-CoT more reliably resolves situated spatial relations by explicitly grounding the observer viewpoint and nearby scene layout.

Idx.	View	Command Examples	L	R	F	U
1		A: While I was carefully wiping down the entrance so I wouldn’t knock over the plant sitting on the books, I saw the tissuebox on the right side of the table in front of the fireplace. B: While I was carefully wiping down the mirror at the entrance so I wouldn’t knock over the plant sitting on the books, I saw the tissuebox on the right side of the table in front of the fireplace. C: The tissuebox was on the right side of the table in front of the fireplace.	✓	✓	✓	✓
2		A: While I was sitting at the table reading a magazine, I saw the dog bed to the right of the window seat. B: While I was sitting in the chair at the table reading a magazine, I saw the dog bed to the right of the window seat. C: The dog bed was to the right of the window seat.	✓	✓	✓	✓
3		A: While I was reaching up to put glasses away above the long white counter, I spotted the mug on the right side of the counter. B: While I was reaching up to put glasses away in the cabinet above the long white counter, I spotted the mug on the right side of the counter. C: The mug was on the right side of the counter.	✓	✓	✓	✓
4		A: While I was washing my hands in the bathroom with a painting on the wall, I saw a roll of toilet paper sitting on top of the toilet tank. B: While I was washing my hands at the sink in the bathroom with a painting on the wall, I saw a roll of toilet paper sitting on top of the toilet tank. C: A roll of toilet paper was sitting on top of the toilet tank.				✓
5		A: While I was washing dishes, I saw a jar of jam on the left side of the stove. B: While I was washing dishes at the sink, I saw a jar of jam on the left side of the stove. C: A jar of jam was on the left side of the stove.	✓

Table 3: Real-world evaluation examples constructed from the house-tour videos Noori (2025) ; Dowling (2026) . Each row corresponds to one set of observer-anchor-target objects. The observer-view image becomes the reference image for C: Given commands. Checkmarks indicate which spatial-direction conditions were evaluated.

Model	Inferred	Stated	Given
GPT-5.4	0.29	0.21	0.41
Qwen3-VL-8B (Transformer)	0.03	0.09	0.24
Gemma-4	0.41	0.38	0.47
Gemini-2.5-Flash	0.09	0.09	0.21
Gemini-Robotics-ER	0.29	0.24	0.24
Qwen3-VL-8B + S-CoT	0.15	0.15	–

Table 4: Manually evaluated success rates (0–1, higher is better) for real-world scene evaluation, pooled over two house-tour videos. Instances without a parseable prediction are counted as failures. The last row applies the S-CoT prompt to Qwen3-VL-8B (Transformer).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Distribution of room counts across the 47 benchmark houses. The red curve shows a fitted Gaussian distribution illustrating variability in environment complexity.

$\bullet$ ArmChair	$\bullet$ GarbageCan
$\bullet$ Bed	$\bullet$ HandTowelHolder
$\bullet$ Chair	$\bullet$ HousePlant
$\bullet$ ClothesDryer	$\bullet$ Microwave
$\bullet$ CoffeeMachine	$\bullet$ Safe
$\bullet$ CoffeeTable	$\bullet$ ShelvingUnit
$\bullet$ CounterTop	$\bullet$ SideTable

Appendix

Table 5: 31 object categories used as observer and anchor objects in the benchmark.

Surface	Candidate Target Objects
Floor	Cloth, Boots, BasketBall, BaseballBat, Dumbbell, DogBed
ArmChair	Book, TeddyBear, Cloth, TissueBox, Newspaper, Mug, Cup, CellPhone, RemoteControl, Watch
Bed	Book, Pillow, TeddyBear, Cloth, TissueBox, Newspaper, CellPhone, RemoteControl, Watch
Chair	Cloth, Book, Pen, Pencil, CellPhone, KeyChain, Watch
CoffeeTable	Book, TissueBox, Newspaper, Mug, Cup, Bottle, RemoteControl, CellPhone, Watch, KeyChain
CounterTop	Mug, Cup, Bottle, Plate, Bowl, Fork, Spoon, Knife, TissueBox

Appendix

Table 6: Examples of surface-specific target object pools. Target candidates are selected according to the physical support surface associated with the sampled spatial relation.

Relation	Anchor	$(x,z)$	$y$
Up	any	$\mathbf{c}$	$y_{\mathrm{top}}+0.2$
Front	any	$\mathbf{b}(\hat{\mathbf{d}})+0.2\,\hat{\mathbf{d}}$	$y_{\mathrm{surf}}+0.2$
Left/Right	$\notin\mathcal{G}$	$\mathbf{b}(\hat{\mathbf{d}})+0.2\,\hat{\mathbf{d}}$	$y_{\mathrm{surf}}+0.2$
Left/Right	$\in\mathcal{G}$ , on anchor	$\mathbf{c}+0.2\,\hat{\mathbf{d}}$	$y_{\mathrm{top}}+0.2$
Left/Right	$\in\mathcal{G}$ , not on anchor	$\mathbf{b}(\hat{\mathbf{d}})+0.2\,\hat{\mathbf{d}}$	$y_{\mathrm{surf}}+0.2$

Appendix

Table 7: Target placement rules for each spatial relation.

Paper name	Model identifier or endpoint	Conditions
MolmoPoint-8B	allenai/MolmoPoint-8B	A, B, C
GPT-5.4	gpt-5.4	A, B, C
Qwen3-VL-8B (Transformer)	Qwen/Qwen3-VL-8B-Instruct	A, B, C
Qwen3-VL-8B (vLLM)	Qwen/Qwen3-VL-8B-Instruct	A, B, C
Qwen3-VL-32B (vLLM)	Qwen/Qwen3-VL-32B-Instruct	A, B, C
InternVL3-38B	OpenGVLab/InternVL3-38B (8-bit quantized)	A, B, C

Appendix

Table 8: Model configurations used in the POVBench experiments. Qwen3-VL-8B is evaluated using both Transformers and vLLM implementations. InternVL3-38B is evaluated with 8-bit quantization. RoboPoint and Llama-3.2-Vision are evaluated only under the Given condition because they do not produce valid responses in the multi-image settings.

Model	Inferred	Stated	Given	Overall
MolmoPoint-8B	97.7	98.5	98.9	98.4
GPT-5.4	97.6	97.6	97.6	97.6
Qwen3-VL-8B (Transformer)	100.0	100.0	100.0	100.0
Qwen3-VL-8B (vLLM)	100.0	100.0	100.0	100.0
Qwen3-VL-32B (vLLM)	100.0	100.0	94.6	98.2
InternVL3-38B	100.0	100.0	100.0	100.0

Appendix

Table 9: Response rates (%) on POVBench. Overall is the average over the conditions each model is evaluated under. A response is counted when the model produces a parseable coordinate prediction.

Explore similar work

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
Grounded 3D-Aware Spatial Vision-Language Modeling
May 28, 2026An-Chieh Cheng, Yang Fu, Yatai Ji +123D Spatial Reasoning Visually Grounded Reasoning
UCSD · NVIDIA · MIT
Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
Aug 24, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Visual Spatial Reasoning Vision-Language Grounding
Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool

Oct 8, 2026, cs.CVJ/K move · Enter open · S save

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang+4

Zhejiang University

Spatial Reasoning Spatial Reasoning Benchmarks Visual Spatial Reasoning VLM Reasoning 3D Spatial Reasoning

Open paper AskPDF arXiv

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.

Abstract

Explore similar work

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Grounded 3D-Aware Spatial Vision-Language Modeling

Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?