An agent that moves must recognise a place from a viewpoint it has never seen. We introduce 4MT-VLM, a dataset of procedurally generated landscapes, each rendered across five stimulus modes that remove appearance cues while holding layout fixed: shape and colour, shape only, colour only, bare terrain peaks with no objects, and a valley viewpoint that puts the peaks on the horizon. The last condition is commonly used in clinics to probe hippocampal function in human patients. We test this benchmark across sixteen different open and closed-source models and report 4AFC performance, a measure which is also used to grade human participants. We observe that models identify a place from the studied viewpoint but lose it once the camera moves, dropping below the 25% chance level at 135° where a human observer scores 85%. Frontier models (Gemini 3.8 Flash, GPT-5.6) answer only 39% and 31% of rotated trials correctly, recovering to 85% and 55% only when distractors are moved more than 30 meters apart. Our benchmark demonstrates that while current VLMs possess rudimentary cognitive maps, their spatial resolution remains fundamentally too coarse to maintain a stable, 3D understanding of the world once the viewpoint changes.
Figures & tables
Figure 1: Top: one place in the five stimulus modes, at one viewpoint. Colour is removed, then shape, then the objects themselves, and c4 moves the camera into the valley so the peaks sit on the horizon. Layout is identical across the five. Middle and bottom: one trial in c0 and one in c4. The study view, then four candidates rendered 135∘ away with appearance resampled, labelled with their layout distance to the target.
Observer
Params
Accuracy
95% CI
Δ=0 gate
Δ≥45
Human
–
86%
[78%, 91%]
100%
82%
InternVL3.5-4B
4B
17%
[11%, 26%]
35%
12%
InternVL3.5-2B
2B
21%
[14%, 30%]
25%
20%
InternVL3.5-1B
1B
23%
[16%, 32%]
35%
20%
InternVL3.5-38B
38B
28%
[20%, 37%]
65%
19%
InternVL3.5-8B
8B
28%
[20%, 37%]
55%
21%
Table 1: Four-alternative forced choice with hard foils (100 trials, chance 25%: 20 unrotated and 80 rotated). The Δ=0 column is the appearance gate. Qwen3-VL-235B (i) is the instruct variant, (t) the thinking one. Rows are ordered by overall accuracy. Bold marks the human row and the best model in each column. Accuracy by mode is reported in Appendix Table 3 .
Figure 2: Left: unrotated against rotated accuracy. An observer working from appearance sits in the lower right; one that cannot identify the scene at all sits in the lower left. Grey points are the models tested but without labels in this figure. Right: the same two quantities against parameter count, on the identical trials, for Qwen2.5-VL at 3B, 7B, 32B and 72B and Qwen3-VL-235B in instruct mode. Grey is unrotated accuracy, which rises with scale while rotated accuracy does not, on hard trials or on simple ones. InternVL3.5 runs the same ladder and lands on top of the Qwen line and is reported in Table 1 .
Observer
Δ=0∘
Δ=45∘
Δ=90∘
Δ=135∘
Δ=180∘
Human
100%
90%
80%
85%
75%
InternVL3.5-4B
35%
20%
20%
5%
5%
InternVL3.5-2B
25%
10%
25%
20%
25%
InternVL3.5-1B
35%
15%
30%
25%
10%
InternVL3.5-38B
65%
15%
15%
25%
20%
Table 2: Accuracy by viewpoint change. The images above the columns are one c0 scene at each viewpoint change, from the same study view. Pooled over the models the floor is Δ=135∘ , below chance, and accuracy recovers at a half turn. Bold marks each model’s own worst viewpoint; rows that tie for worst are left unmarked.
Figure 3: Left: Qwen2.5-VL-32B under six instruction styles, rotated trials only, with the instruction used everywhere else highlighted. The styles are, from the top, "imagine a plan view", "pick an anchor landmark", "rotate the scene mentally", "eliminate alternatives", plain chain of thought, and chain of thought with the viewpoint change asserted. None reaches chance. Right: accuracy against the number of landmarks in a scene, one panel per model. The shaded band is the gap between identifying a place and recognising it after a viewpoint change. Adding landmarks raises the unrotated line in all three models and closes the gap in only one. Landmark count and layout separation cannot be varied independently: the median distractor sits at 0.9, 7.6, 23.2 and 28.6 m for one, two, four and six landmarks.
Table 3: Rotated-trial accuracy by stimulus mode, 16 trials per cell, with one place shown in each mode above its column. c0 shape and colour, c1 shape only, c2 colour only, c3 bare peaks, c4 valley. Rows are ordered by overall accuracy. Bold marks the human row and, for each model, its weakest mode
Figure 4: Left: do a pair of observers agree on which trials are solvable? Cohen’s κ on trial-level correctness; higher values mean a pair succeeds and fails on the same trials. The human observer is in the first row, with models ordered by mutual similarity. Right: when an observer is wrong, which distractor does it pick? Distractors are ranked by layout distance to the target, and the dashed lines mark thirds, which is what errors independent of layout geometry would give.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
We push the frontier of large-scale spatial intelligence in Vision-Language Models (VLMs) and introduce the first benchmark that probes geographical layout understanding from real-world videos, spanning up to 1km distances. Inspired by the cognitive science literature, we evaluate models against the hierarchical stages of human spatial awareness: anchoring via landmarks, connecting them through routes, and integrating these into global mental maps. Extensive experiments reveal a fundamental divergence in how current AI models process spatial information. Instead of utilising true path integration or forming geometric survey knowledge, we find that VLMs rely almost entirely on 2D visual recognition and text-matching to bypass complex spatial reasoning. The benchmark is publicly available at https://perception-test-challenge.github.io/kilometervision.html.
Aravindh Mahendran, Michael King, Matthew Koichi Grimes +12
Google DeepMind, Berlin, Germany · Google DeepMind, London, UK · Princeton University, Princeton, USA +3
Recent benchmarks for VLMs largely assess single- or limited-view perception, leaving untested the core cognitive ability to integrate observations across viewpoints into a coherent, world-centric (allocentric) 3D mental model. We introduce MultiView-Bench, a diagnostic benchmark expressly designed to evaluate multi-view integration for holistic 3D scene comprehension. Unlike existing datasets that focus on pixel-level mapping or camera-relative navigation, MultiView-Bench requires models to decouple object positioning from transient perspectives and ground them in a fixed global coordinate system. This capability serves as a prerequisite for VLMs before being deployed for downstream tasks such as mechanical part assembly. Our systematic evaluation of frontier VLMs reveals consistent failure modes: strong performance on 2D planar relations from a single image, but marked difficulty with 3D spatial relations and with aggregating information across views. We further identify biases in VLMs, such as struggles with unconventional axis directions and sensitivity to object colorways and texture variations. Acknowledging these limitations, we propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints, perceives, and fuses multi-view evidence, improving diverse base models on MultiView-Bench even under a strict budget-matched comparison (and by 3-5x for the full agent).
Modern Vision-Language Models (VLMs) perform well above the human baseline in image geolocalization, a task critically important in disaster response, OSINT verification, and location privacy. However, most efforts to study AI behavior on the task remain limited to static image-based retrieval, classification, and predictions. We argue that faithful recreation of the task should involve embodied navigation, where a multimodal agent autonomously explores its surroundings to gather observations before submitting a prediction. To this end, we introduce GeoAgent, an agentic environment-based benchmark that requires agents to navigate Street View environments to refine their geolocalization through sequential reasoning. Our analysis shows that modern VLMs struggle to discern regional patterns while succeeding at country- and continent-level predictions. When compared to static image-based baselines, agentic navigation significantly improves accuracy across established metrics. We also note severe bias in a developed/developing region context across frontier model architectures and poor self-improvement capabilities given incorrect priors. Overall, our work establishes the challenges of embodied navigation and geospatial reasoning. We publicly release our code and the GeoAgent environment: https://geoagent-benchmark.github.io