Multimodal Language Models Cannot Spot Spatial Inconsistencies

Om Khangaonkar; Hadi J. Rad; Hamed Pirsiavash

cs.CVApr 1, 2026

Multimodal Language Models Cannot Spot Spatial Inconsistencies

Authors: Om Khangaonkar, Hadi J. Rad, Hamed Pirsiavash

Organizations: University of California, Davis · Shell · TU Delft

Abstract

Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances, multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.

Figures & tables

Figure 1: Synthesizing spatially inconsistent image pairs from multi-view data. Given three views $(V_{1},V_{2},V_{3})$ of the same static scene, we (1) select an object $O$ visible in all views, (2) Erase $O$ in view $V_{2}$ and inpaint to obtain a clean background, and (3) paste the instance of $O$ from view $V_{3}$ back into $V_{2}$ at its original location. Because $V_{3}$ is captured from a different camera pose, the pasted object has an appearance that is incompatible with the 3D geometry implied by $V_{1}\rightarrow V_{2}$ , while all other objects remain consistent. This yields realistic image pairs with controlled spatial violations and no manual annotation. We label objects in $V_{1}$ for our forced-choice evaluation.

Panel of Figure 2 — Figure 2: Example spatial inconsistencies. AI labels are ordered as GPT-5 (LR) , Gemini 2.5 Pro (MR) and Qwen3-VL 8B Instruct . Zoom in to see labels in detail.

Model	Overall	Model	Overall
Gemma 3 12B	8.5	Random Chance	7.9
Idefics3 8B	8.9	Human	84.8
Idefics2 8B	11.9	GPT-5 Nano	15.3
Qwen2.5-VL 7B	15.9	Gemini 2.5 Flash	17.6
InternVL 3.5 8B	16.1	GPT-4o	19.0
LLaVA OneVision 1.5 8B	16.6	Gemini 2.5 Pro ( HR )	28.9

Table 1: Overall accuracy (%) on the spatial inconsistency identification task. We report overall accuracy for a random-choice baseline, humans, and a range of multimodal language models. We find that humans substantially outperform all tested models. Interestingly, higher reasoning budget does not necessarily yield higher accuracy. Ensembling open source models also allows them to perform at the level of large-scale proprietary models.

Figure 3: Model accuracy varies across scene attributes, but much less across the number of labels per pair. Left: We report accuracy for identifying the single spatially inconsistent object, stratified by inconsistent object depth (close/medium/far), average pair brightness (dark/medium/bright), and the augmented object’s physical plausibility. While humans remain comparatively robust across conditions, models can show substantial sensitivity to changes in the pair’s scene composition. Right: We partition the dataset based on the number of labels per pair and report accuracy. Interestingly, this has a minor effect on humans but not much of one on models. Dashed line represents random chance.

Panel of Figure 4 — Figure 4: Accuracy varies greatly across the inconsistent object or pair scene categories. While humans are relatively consistent across all settings, the models show large amounts of variance. This suggests that their 3D understanding is brittle across our diverse visual world. Dashed line represents random chance.

	Our	Artifacts only	No inconsistency or arti-
Model	Benchmark	( $V_{2}$ self-paste)	facts ( $V_{2}$ no change)
Random Chance	7.9	7.9	7.9
Qwen3-VL 4B	24.7	19.5	20.7
Qwen3-VL 8B	27.6	21.3	22.0
Gemini 2.5 Pro (MR)	29.4	13.8	15.3
GPT-5 (LR)	34.2	15.1	13.7

Table 2: Models do not succeed by detecting inpainting artifacts alone. We compare performance on our full benchmark against two controls: a self-pasting baseline that preserves inpainting artifacts while removing the inconsistency, and a no-change baseline that contains neither artifacts nor inconsistency. Across all models, self-pasting performs similarly to the no-change baseline, indicating that artifact cues by themselves do not explain benchmark performance.

	# of self-pasted objects
Model	None	1	3	5	10	all
Qwen3-VL 4B Instruct	24.7	25.0	24.9	25.2	25.0	25.0
Qwen3-VL 8B Instruct	27.6	27.3	26.7	26.0	25.0	25.4
Gemini 2.5 Pro (MR)	29.4	28.1	29.8	28.9	26.7	24.7
GPT-5 (LR)	34.2	31.7	34.0	33.0	33.5	30.2

Table 3: Self-pasting additional objects does not make our task harder. Performance remains largely stable even when many or all objects are self-pasted, suggesting that models are looking for 3D structure and the task is not confounded by artifacts.

	Inpainting bbox expansion
Model	5%	0%	10%	25%	50%	75%	100%
Qwen3-VL 4B Instruct	24.7	25.9	26.3	27.6	31.9	35.8	37.6
Qwen3-VL 8B Instruct	27.6	27.8	27.8	28.8	34.8	39.5	44.1
Gemini 2.5 Pro (MR)	29.4	30.3	28.2	29.1	39.5	43.2	44.9
GPT-5 (LR)	34.2	31.7	30.6	34.0	43.3	42.8	48.0

Table 4: Inpainting artifacts are too small to confound evaluation . We gradually increase the bounding box for inpainting until the artifacts are too large to notice (which occurs around 25-50%). We use 5% when generating pairs, which is far smaller than this threshold.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Object Depth			Scene Lighting			Plausibility
	Close	Medium	Far	Dark	Medium	Bright	IP	P
Random Chance	7.2	8.0	8.4	7.7	8.1	7.8	7.9	7.7
Human	83.3	87.3	83.8	82.3	85.4	86.8	86.3	80.1
Gemma 3 12B	13.2	7.3	4.9	7.8	11.2	6.3	9.4	5.4
Idefics3 8B	8.8	10.7	7.3	8.8	9.8	8.3	7.7	12.8
Idefics2 8B	9.8	12.2	13.7	12.2	13.2	10.2	12.4	10.1

Appendix

Table 5: Performance breakdown (% accuracy) by depth, lighting, and physical plausibility. We report the full results for each model, of which a subset was shown in Figure 3 .

Model	Chair	Window	Picture	Sofa	Cabinet	Door
Random Chance	8.4	7.0	7.4	8.3	7.0	7.6
Human	86.8	82.5	86.4	84.8	88.1	85.7
Gemma 3 12B	15.3	4.1	8.1	7.3	8.3	11.4
Idefics3 8B	8.5	1.4	4.8	17.1	11.1	11.4
Idefics2 8B	4.2	19.2	19.4	7.3	13.9	5.7
GPT-5 Nano	13.6	4.1	19.4	12.2	11.1	22.9

Appendix

Table 6: Accuracy by inconsistent object class. We report the full results for each model, of which a subset was shown in Figure 5 .

Model	Lounge	Kitchen	Study	Hallway	Bedroom
Random Chance	8.4	6.3	7.1	8.7	8.1
Human	88.9	83.3	81.0	85.9	85.7
Gemma 3 12B	5.6	8.5	17.4	9.0	4.8
Idefics3 8B	8.8	6.8	9.2	9.0	7.9
Idefics2 8B	12.0	9.4	16.5	7.5	11.1
GPT-5 Nano	9.6	17.1	21.1	11.9	12.7

Appendix

Table 7: Accuracy by scene class. We report the full results for each model, of which a subset was shown in Figure 5 .

Model	5–10	11–15	16–20	21–26
Random Chance	12.3	7.9	5.7	4.1
Human	86.5	86.8	85.6	78.5
Gemma 3 12B	12.1	6.5	7.3	7.5
Idefics3 8B	13.3	9.7	6.5	4.5
Idefics2 8B	16.2	9.2	11.4	10.4
GPT-5 Nano	19.1	13.0	18.7	10.4

Appendix

Table 8: Accuracy by number of labels. We report the full results for each model, of which a subset was shown in Figure 3 .

Model	Accuracy on DL3DV inconsistencies
Random Chance	18.4
Qwen3-VL 4B	47.7
Qwen3-VL 8B	57.7
Gemini 2.5 Pro (MR)	52.9
GPT-5 (LR)	52.4

Appendix

Table 9: When we scale up to real-world data, our task is still challenging for models.

Model	Base Model	Hypersim	DL3DV
Hypersim Inconsistencies	24.7	66.8 $\pm$ 1.5	55.2
Multiview Reasoning	37.6	39.9 $\pm$ 1.4	43.6
Visual Correspondence	80.2	81.7 $\pm$ 0.7	82.2
Relative Depth	84.7	86.9 $\pm$ 1.1	87.7

Appendix

Table 10: We show that Qwen3-VL 4B finetuned with additional scene diversity perform on par or better than our initial models at 3D perception and perform well on the Hypersim evaluation set.

Model	Base Model	Finetuned
Our Benchmark, disjoint set	24.7	69.2

Appendix

Table 11: Fine-tuning on our spatial inconsistency task yields large gains on a disjoint test split. Accuracy (%) of the base vision–language model versus the same model fine-tuned on our benchmark, evaluated on a held-out set of image pairs with no scene overlap. Performance jumps from 24.7% to 69.2%, showing that the task is learnable and that targeted training substantially improves spatial reasoning on new scenes.

Model	Pairwise	Pearson	Kendall
GPT–4o	50.1	-0.14	0.32

Appendix

Table 12: VQAScore evaluation of GPT–4o on our spatial inconsistency benchmark. We report the pairwise accuracy (%) of VQAScore ( Lin et al., 2024 ) when asked to score both the physically consistent image and the manipulated one, along with Pearson and Kendall correlations between VQAScore and human accuracy across image pairs. The near-chance pairwise score and weak correlations indicate that the SOTA VQA-based judge poorly aligns with human assessments of spatial consistency.

Multi-View Consistency

Multi-view consistency studies how visual models keep geometry, appearance, identity, and predictions aligned across different views of the same scene.

See all 192 papers

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning 3D Scene Representation
KAIST AI · ETH Zürich · Google +2
Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning
Apr 8, 2026Jiahua Chen, Qihong Tang, Weinong Wang +13D Spatial Reasoning VLM Reasoning
Tsinghua University · Nanjing University · Tencent
What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
Oct 13, 2026Xiaohu Huang, Yumin Guo, Kai HanMultimodal Spatial Reasoning Multimodal Large Language Models
The University of Hong Kong

Sep 29, 2026J/K move · Enter open · S save

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han+9

KAIST AI · ETH Zürich · Google+2

3D Spatial Reasoning 3D Scene Representation 3D Scene Understanding 3D VLMs Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Abstract

Multi-View Consistency

Explore similar work

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?