EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery

Zelin Xu; Yupu Zhang; Saugat Adhikari; Saiful Islam; Tingsong Xiao; Zibo Liu; Shigang Chen; Da Yan

cs.CVFeb 17, 2026

EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery

Authors: Zelin Xu, Yupu Zhang, Saugat Adhikari, Saiful Islam, Tingsong Xiao, Zibo Liu, Shigang Chen, Da Yan,

Organizations: Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA · Department of Computer Science, Indiana University Bloomington, Bloomington, IN, USA.

Abstract

Benchmarking spatial reasoning in multimodal large language models (MLLMs) has attracted growing interest in computer vision due to its importance for embodied AI and other agentic systems that require precise interaction with the physical world. However, spatial reasoning on Earth imagery has lagged behind, as it uniquely involves grounding objects in georeferenced images and quantitatively reasoning about distances, directions, and topological relations using both visual cues and vector geometry coordinates (e.g., 2D bounding boxes, polylines, and polygons). Existing benchmarks for Earth imagery primarily focus on 2D spatial grounding, image captioning, and coarse spatial relations (e.g., simple directional or proximity cues). They lack support for quantitative direction and distance reasoning, systematic topological relations, and complex object geometries beyond bounding boxes. To fill this gap, we propose EarthSpatialBench, a comprehensive benchmark for evaluating spatial reasoning in MLLMs on Earth imagery. The benchmark contains over 325K question-answer pairs spanning: (1) qualitative and quantitative reasoning about spatial distance and direction; (2) systematic topological relations; (3) single-object queries, object-pair queries, and compositional aggregate group queries; and (4) object references expressed via textual descriptions, visual overlays, and explicit geometry coordinates, including 2D bounding boxes, polylines, and polygons. We conducted extensive experiments on both open-source and proprietary models to identify limitations in the spatial reasoning of MLLMs.

Figures & tables

Datasets	Number of QA Pairs	Geometry Type	Spatial Reasoning
Datasets	Number of QA Pairs	Geometry Type	Distance	Direction	Topological
EarthVQA ( Wang et al., 2024 )	209K	BBox	$\times$	$\times$	$\times$
GeoChatSet ( Kuckreja et al., 2024 )	318K	BBox	$\times$	$\times$	$\times$
VRSBench ( Li et al., 2024b )	123K	BBox	$\times$	$\times$	$\times$
SkyEyeGPT ( Zhan et al., 2025 )	968K	BBox	$\times$	$\times$	$\times$
XLRS-Bench ( Wang et al., 2025a )	46K	BBox	$\times$	$\triangle$	$\times$

Table 1: A comparison of existing benchmark datasets for MLLM on Earth imagery. (

\checkmark

”Fully Benchmarked”;

\triangle

”Partially Benchmarked”;

\times

”Not Benchmarked”)

Figure 1: Overview of EarthSpatialBench. The benchmark spans diverse spatial relations evaluated over multiple geometry types, object representation modalities, and question formats, enabling comprehensive assessment of spatial reasoning. High-resolution Earth imagery and geometric annotations are collected from SatlasPretrain. Spatial relations (distance, direction, topology) are computed using standard GIS procedures and combined with curated question templates to generate diverse reasoning tasks. A multi-stage quality control process ensures geometric accuracy, semantic clarity, and reliable ground-truth annotations.

Property	Value
Source dataset	SatlasPretrain ( Bastani et al., 2023 )
Sensor type	Optical RGB
Image resolution	1m
Image size	512 $\times$ 512
Total images	21K
Total QA pairs	325K

Table 2: Overview of EarthSpatialBench.

Panel of Figure 2 — Figure 2: (a) Distribution of all 23 geometric object types in EarthSpatialBench (log scale). The dataset includes 6 small-object bounding-box classes (building, dam, parking lot, pier, power substation, helipad), 7 polyline classes (airport runway, airport taxiway, raceway, railway, river, road, track), and 10 large-region polygon classes (airport, crop, landfill, park, quarry, solar farm, water park, theme park, stadium, power plant). (b) Distribution of the total number of annotated objects per image on a log–log scale, showing the wide variation in scene complexity across images. (c) Distribution of question counts across spatial relation categories (distance, direction, topology) and question formats (choice-based, quantitative, localization).

Model	Exact Distance		Count		Localization		Binary Classification
Model	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	F1 $\uparrow$	IoU $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Qwen3-VL-I-30B	127.93	168.31	1.28	2.06	0.25	0.18	0.58	0.56
Qwen3-VL-I-30B-CoT	124.78	169.16	1.19	1.41	0.16	0.16	0.53	0.34
Qwen3-VL-T-30B	114.87	158.92	1.02	1.77	0.25	0.19	0.71	0.48
Qwen3-VL-T-30B-CoT	121.17	158.39	1.33	1.68	0.26	0.20	0.56	0.34
InternVL3.5-38B	144.21	189.73	0.89	1.19	0.23	0.19	0.59	0.54

Table 3: Performance on Distance reasoning tasks.

Model	Exact Angle		Count		Localization		8-Direction Classification		Binary Classification
Model	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	F1 $\uparrow$	IoU $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Qwen3-VL-I-30B	117.20	145.86	2.88	6.37	0.24	0.16	0.35	0.31	0.67	0.67
Qwen3-VL-I-30B-CoT	80.31	130.91	1.74	5.21	0.04	0.05	0.34	0.34	0.57	0.38
Qwen3-VL-T-30B	117.98	140.94	1.64	2.90	0.21	0.13	0.26	0.26	0.71	0.71
Qwen3-VL-T-30B-CoT	54.99	96.28	2.24	6.37	0.16	0.11	0.45	0.38	0.70	0.46
InternVL3.5-38B	126.31	158.25	1.55	3.14	0.16	0.11	0.30	0.29	0.64	0.64

Table 4: Performance on Direction reasoning tasks.

Model	Count				Localization				Binary Classification
Model	Intersect		Within		Intersect		Within		Intersect		Within
	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	F1 $\uparrow$	IoU $\uparrow$	F1 $\uparrow$	IoU $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Qwen3-VL-I-30B	1.14	1.82	2.45	4.30	0.24	0.19	0.13	0.15	0.68	0.63	0.90	0.71
Qwen3-VL-I-30B-CoT	1.10	1.98	2.83	6.55	0.19	0.17	0.15	0.13	0.71	0.69	0.94	0.77
Qwen3-VL-T-30B	1.07	1.88	2.79	6.54	0.21	0.18	0.21	0.20	0.73	0.72	0.91	0.68
Qwen3-VL-T-30B-CoT	1.07	2.09	3.06	6.84	0.19	0.18	0.12	0.07	0.72	0.72	0.94	0.68

Table 5: Performance on Topological reasoning tasks.

Figure 3: Performance of MLLMs on four grounding variants.

Panel of Figure 4 — Figure 4: Performance differences between visual-overlay-based and coordinate-based object references. (a) Change in MAE for counting. (b) Change in F1 score for localization.

Figure 5: MAE of angle estimation under different object representation settings.

Figure 6: MAE of distance estimation across different geometry type combinations.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 156 papers

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
Oct 9, 2026Zheng Ren, Yanning Wang, Ziheng Huang +4Geospatial Reasoning Benchmarks Geospatial Reasoning
Urban Governance and Design (UGOD) Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Spatial Reasoning Multilingual QA
Data and Web Science Group, University of Mannheim, Germany
SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models
Aug 3, 2026Hai Nguyen, Tung Vu, Cong TranSpatial Reasoning Benchmarks 3D Spatial Reasoning
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam

Oct 9, 2026J/K move · Enter open · S save

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

Zheng Ren, Yanning Wang, Ziheng Huang, Fei Huang+3

Urban Governance and Design (UGOD) Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Geospatial Reasoning Benchmarks Geospatial Reasoning Spatial Reasoning Spatial Reasoning Benchmarks VLM Reasoning

Open paper AskPDF arXiv

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models