Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

Zheng Ren; Yanning Wang; Ziheng Huang; Fei Huang; Shiqi Wang; Ziqian Ye; Yang Yue

cs.AIOct 9, 2026

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

Authors: Zheng Ren, Yanning Wang, Ziheng Huang, Fei Huang, Shiqi Wang, Ziqian Ye, Yang Yue

Organizations: Urban Governance and Design (UGOD) Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Abstract

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.

Figures & tables

Figure 1: Overview of GeoSpatialBench. QA and spatial editing are constructed from urban vector data across six cities and 28 spatial relations. Both tasks support GeoJSON, image, and combined inputs. QA is evaluated by answer accuracy. GeoJSON and Together editing use the Constraint Satisfaction Rate (CSR), while map-image editing uses a raster-tolerant success rate.

Figure 2: Average QA and spatial editing performance across models and input modalities. (a) QA accuracy averaged over open-ended and multiple-choice settings. (b) Spatial editing success. GeoJSON and Together editing use strict CSR, whereas Image editing uses raster-tolerant success.

Figure 3: Performance across spatial relations under GeoJSON input. (a) Mean performance across the four relation families for open-ended QA, multiple-choice QA, and spatial editing. (b) Relation-level multiple-choice QA accuracy for each model across the 28 spatial relations. Metric relations are generally more challenging, while QA and spatial editing show different patterns across relation families.

Figure 4: Representative QA cases across the four spatial relation families. Map images often preserve coarse spatial judgments but are less reliable for precise boundary and metric reasoning.

Figure 5: Representative spatial editing cases across four relation families, illustrating failures in geometric execution and constraint satisfaction.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	GeoJSON			Map image			Together
Model	Open	MCQ-4	Edit	Open	MCQ-4	Edit	Open	MCQ-4	Edit
GPT-6 Astra	96.36	96.64	97.60	60.57	73.12	44.81	94.86	95.86	98.71
KIMI K3	89.52	91.88	85.10	51.43	59.29	39.29	92.86	93.43	85.29
GPT-5.6 Sol	78.12	85.43	72.90	47.40	56.02	30.12	77.14	84.29	70.86
MiMo-V2.6-Pro	62.98	71.71	42.98	39.19	51.14	16.50	59.55	70.93	51.38
Qwen3.8-27B	52.45	58.64	54.38	38.43	42.00	18.29	53.52	55.93	52.19

Appendix

Table 1: Overall performance (%; higher is better), ordered by GeoJSON open-ended QA. Each model uses 4,200 instances across six cities. Image editing reports raster-tolerant success; GeoJSON and Together editing report strict CSR. Bold marks the highest reported score in each column.

Model	Stockholm	Paris	Nairobi	Hong Kong	Wuhan	Detroit	Overall	Range
GPT-6 Astra	95.57	96.14	96.14	96.71	96.86	96.71	96.36	1.29
KIMI K3	89.29	90.00	89.86	89.43	89.00	89.57	89.52	1.00
GPT-5.6 Sol	77.86	79.57	79.57	77.57	76.57	77.57	78.12	3.00
MiMo-V2.6-Pro	62.57	64.14	62.14	64.43	61.43	63.14	62.98	3.00
Qwen3.8-27B	51.14	53.14	53.00	52.43	51.71	53.29	52.45	2.14
DeepSeek-V4.1-Flash	48.43	49.00	49.14	48.71	49.86	51.86	49.50	3.43

Appendix

Table 2: Open-ended GeoJSON QA across six cities (%; higher is better), ordered by pooled accuracy. Each city has 700 questions per model. Overall pools 4,200 questions; Range is the maximum minus minimum city score in percentage points.

Figure 6: Error rates by spatial relation, input, and task (%; lower is better).

	GeoJSON			Map image
Family	Open QA	MCQ-4	Editing	Open QA	MCQ-4	Editing
Topology	25.61	22.09	34.86	50.30	46.27	83.32
Direction	44.68	38.04	36.64	55.76	44.21	81.37
Metric	61.40	46.57	57.09	84.87	63.69	86.53
Ordering	28.14	30.59	52.67	45.43	50.44	66.87

Appendix

Table 3: Error rates by relation family (%; lower is better), pooled over the nine reported models. GeoJSON and Map Image use six cities for every model.

			GeoJSON			Map image
ID	Relation	Family	Open QA	MCQ-4	Editing	Open QA	MCQ-4	Editing
T01	Equality	Topology	13.11	1.63	3.26	82.78	68.73	92.82
T02	Disjointness	Topology	7.33	8.37	47.26	13.47	23.43	32.90
T03	Intersection	Topology	5.78	16.67	29.48	19.59	27.43	82.53
T04	Touching	Topology	20.22	7.33	71.19	38.20	25.47	92.98
T05	Crossing	Topology	22.44	21.48	32.44	25.47	14.04	84.08

Appendix

Table 4: Error rates for the 28 spatial relations (%; lower is better).

Geospatial Reasoning Benchmarks

See all 31 papers

Feb 17, 2026

EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery

Benchmarking spatial reasoning in multimodal large language models (MLLMs) has attracted growing interest in computer vision due to its importance for embodied AI and other agentic systems that require precise interaction with the physical world. However, spatial reasoning on Earth imagery has lagged behind, as it uniquely involves grounding objects in georeferenced images and quantitatively reasoning about distances, directions, and topological relations using both visual cues and vector geometry coordinates (e.g., 2D bounding boxes, polylines, and polygons). Existing benchmarks for Earth imagery primarily focus on 2D spatial grounding, image captioning, and coarse spatial relations (e.g., simple directional or proximity cues). They lack support for quantitative direction and distance reasoning, systematic topological relations, and complex object geometries beyond bounding boxes. To fill this gap, we propose EarthSpatialBench, a comprehensive benchmark for evaluating spatial reasoning in MLLMs on Earth imagery. The benchmark contains over 325K question-answer pairs spanning: (1) qualitative and quantitative reasoning about spatial distance and direction; (2) systematic topological relations; (3) single-object queries, object-pair queries, and compositional aggregate group queries; and (4) object references expressed via textual descriptions, visual overlays, and explicit geometry coordinates, including 2D bounding boxes, polylines, and polygons. We conducted extensive experiments on both open-source and proprietary models to identify limitations in the spatial reasoning of MLLMs.

Zelin Xu, Yupu Zhang, Saugat Adhikari +6

Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA · Department of Computer Science, Indiana University Bloomington, Bloomington, IN, USA.

Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Multimodal Spatial Reasoning Geospatial Reasoning VLM Reasoning Remote Sensing VQA

Ask PDF

Aug 4, 2026

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.

Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1

Data and Web Science Group, University of Mannheim, Germany

Spatial Reasoning Multilingual QA Multilingual Language Model Evaluation Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Jurisdictional Bias in Language Models+3

Ask PDF

Jun 3, 2026

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely symbolic, arising from pattern matching over spatial language rather than true geometric reasoning over space. Because LLMs operate on discrete tokens, they lack native support for continuous spatial representations, explicit geometric computation, and structured spatial operators. To address this limitation, we introduce the Spatial Language Model (SLM), the first multimodal LLM that treats location information as a first-class modality and enables geometric spatial reasoning within the model's inference process. SLM directly operates on learned spatial representations rather than textual descriptions of spatial relations. To support effective training, we construct a Spatial Instruction Dataset that aligns spatial representations, atomic geometric operations, and natural language instructions. We further propose a new benchmark named SpatialEval, which is designed to evaluate spatial reasoning across attributes, distance, topology, and relative-position tasks. Extensive experiments show that SLM significantly outperforms existing LLM-based approaches that rely on symbolic reasoning via prompt engineering or textual abstraction, demonstrating the benefits of integrating geometric spatial representations for robust spatial reasoning. Our instruction dataset, evaluation benchmark, model training codes, and models' checkpoints can be found at: \hyperlink{https://github.com/chuchen2017/SLM}{https://github.com/chuchen2017/SLM}.

Chen Chu, Bita Azarijoo, Li Xiong +2

University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA

Spatial Reasoning Benchmarks Multimodal Large Language Models Geometric Representation Learning Spatial Reasoning Spatial Reasoning in Language Models

Ask PDF

Abstract

Geospatial Reasoning Benchmarks

Explore similar work

EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models