SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery

Peter A. Massih; Eric Cosatto; Martin Renqiang Min

cs.CVJan 19, 2026

SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery

Authors: Peter A. Massih, Eric Cosatto, Martin Renqiang Min

Organizations: Department of Machine Learning, NEC Laboratories America · EPFL

Abstract

Vision-Language Models (VLMs) describe satellite images fluently but struggle with the quantitative spatial reasoning that remote sensing (RS) analysts perform routinely, such as counting buildings, measuring forest area in hectares and computing distances in meters. We introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 questions over 587 images from four sources at 0.3 to 1.0 m ground sampling distance, covering 24 question types in three difficulty tiers. Answers are computed from curated segmentation masks or, for 50 questions, from human consensus, and numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers, so that modular systems can be evaluated using reference rather than predicted masks, and 75,687 training question-answer pairs on disjoint images (SQuID-Train). Among 15 evaluated systems, the best zero-shot end-to-end VLM reaches 28.1%, and RS-specialized VLMs score lower. As a baseline, we propose QASR (Quantitative Agentic Spatial Reasoning), in which an LLM writes a program that segments the image and measures the masks with geometric operators. QASR reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks instead of predicted masks raises accuracy from 42.2% to 82.6%, so segmentation is its main source of error. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%, close to QASR. Multi-condition questions (Tier 3) remain the hardest in every setting, at 62.0% for QASR with reference masks and 13.2% after fine-tuning. These results leave clear room for end-to-end models that learn metric spatial reasoning, and SQuID provides the data to train and evaluate them.

Figures & tables

Source	GSD	Classes	Images	Questions	Answers	Terms
DeepGlobe [ 9 ]	0.5 m	6	174	612	masks	DeepGlobe terms
EarthVQA [ 44 ]	0.3 m	5	364	1,241	masks	EarthVQA terms
Solar [ 15 ]	0.3 m	1	35	97	masks	CC BY 4.0
NAIP [ 43 ]	1.0 m	7	14	50	human	Public domain
Total			587	2,000

Table 1: SQuID sources. Answers are computed from reference masks, except for NAIP, where they come from human consensus.

Panel of Figure 1 — Figure 1: SQuID evaluation modes. (a) An end-to-end system maps the image, question and GSD to an answer. (b) A modular system runs a generated program on predicted masks. (c) The same program is run again on reference masks, with no new LLM call. On the 1,950 mask-derived questions, this corrects 814 of the 1,127 QASR errors. With the same reference masks, Llama-3.1-8B programs reach 67.1%, compared with 82.6% for GPT-5. Scores in (a) and (b) use all 2,000 questions, while (c) uses the 1,950 questions with reference masks.

Question	GSD	Answer	Range
Tier 1: Basic Quantification
What percentage of the image is covered by barren land?	0.3 m	33.1%	[31.4, 34.9]
How many buildings are there? When counting, ignore buildings smaller than 0.01 hectares.	0.3 m	4	[3, 5]
Is there more water than barren land in this image?	0.3 m	yes	exact
Tier 2: Spatial Relationships
What percentage of the urban area is within 500 m of vegetation?	0.5 m	34.35%	[32.1, 36.6]

Table 2: Representative SQuID questions by tier. Tolerance ranges are informed by human annotations, and the supplementary material gives one question for each of the 24 types.

Panel of Figure 2 — Figure 2: Benchmark images with example questions, reference answers and tolerance ranges.

Panel of Figure 3 — Figure 3: SQuID dataset composition. (A) Three difficulty tiers. (B) Four data sources at 0.3 to 1.0 m GSD. (C) Top 5 question types and the share of all other types. Right: inter-rater reliability on the 47 numeric questions, with 10 annotations per question. The three categorical questions are excluded from this table.

Panel of Figure 4 — Figure 4: ConvNeXt-UNet segmentation. Input image (left) and combined output (right), with forest in dark green, agriculture in yellow, grass in light green, barren land in orange, water in blue, solar panels in dark blue and buildings in magenta.

Method		Base LLM	Segmentation	$\uparrow$ Accuracy (%)			$\uparrow$ Overall (%)
Method		Base LLM	Segmentation	Tier 1	Tier 2	Tier 3	$\uparrow$ Overall (%)
Modular, predicted masks	QASR-A	GPT-5	ConvNeXt	53.52	54.06	18.84	42.00
	QASR-B	gpt-oss-120B	ConvNeXt	44.79	45.94	6.08	32.10
	QASR-C	GPT-5	DINOv3	50.99	47.40	12.46	36.90
	QASR-D	Llama-3.1-8B	ConvNeXt	39.86	41.88	5.79	29.00
	QASR-E	GPT-5	RemoteSAM	23.52	38.64	6.08	22.30

Table 3: Accuracy (%) on SQuID by tier. Among the rows evaluated on the full benchmark, the best result is in bold and the second best is underlined .

Predicted masks	Reference masks	GPT-5	Llama-3.1-8B
Correct	Correct	796	533
Correct	Incorrect	27	34
Incorrect	Correct	814	776
Incorrect	Incorrect	313	607

Table 4: Outcome of each of the 1,950 mask-derived questions when the same QASR programs run on predicted and on reference masks.

Panel of Figure 5 — Figure 5: Left. Accuracy of QASR (GPT-5 + ConvNeXt) grouped by the per-image IoU between predicted and reference masks. QASR uses predicted masks in this analysis. Right. Accuracy advantage of QASR over GPT-5 used end to end, in percentage points, as numeric tolerance ranges are widened by factors from 1.0 to 2.0. The gap grows from 13.9 to 17.5 points.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Encoder	ConvNeXt, pretrained on ImageNet
Decoder	U-Net
Parameters	About 180M per model
Loss	Cross-entropy
Optimizer	Adam
Learning rate	$10^{-4}$

Appendix

Table 5: ConvNeXt-UNet training configuration. The same architecture is used for all four datasets.

Topic	Type	Source classes
urban	semantic	DeepGlobe urban, EarthVQA road, EarthVQA building
forest	semantic	DeepGlobe forest, EarthVQA forest
agric	semantic	DeepGlobe agriculture, EarthVQA agriculture
grass	semantic	DeepGlobe rangeland
barren	semantic	DeepGlobe barren, EarthVQA barren
water	semantic	DeepGlobe water, EarthVQA water

Appendix

Table 6: Topics available to QASR with the ConvNeXt models. When two models predict a class, the topic takes the maximum of their logits. The urban topic combines three source classes.

Training configuration		Results
Backbone	DINOv3 ViT-L/16, frozen	Best mIoU	57.58%
Pretraining	SAT-493M, about 300M parameters	Pixel accuracy	73.50%
Decoder	Adapter and Mask2Former	Best epoch	60 of 200
	(56.6M parameters)	Training time	6.2 hours
Hidden dimension	256
Batch size	20 (10 per GPU)	IoU per class

Appendix

Table 7: DINOv3 + Mask2Former configuration and results on the EarthVQA validation split.

Method	Segmentation	Counting	Other tasks	Overall
GPT-5, end to end		29.6%	57.3%	48.2%
QASR with GPT-5	SAM-3	15.8%	57.3%	43.7%
QASR with GPT-5	RemoteSAM	15.6%	57.3%	43.6%

Appendix

Table 8: Cross-domain evaluation on GEOBench-VLM ( Single split, 3,211 five-way multiple-choice questions). For QASR, the tasks other than counting use GPT-5 end to end.

Panel of Figure 6 — Figure 6: Annotation interface built on Turkle [ 14 ] . (a) Annotators select the grid cells that cover a land-cover class to answer percentage questions, with a grid adjustable from 10 $\times$ 10 to 320 $\times$ 320 cells. (b) A distance ruler supports proximity questions. Ten annotators used this interface to produce 500 annotations for 50 questions over 14 NAIP images at 1.0 m GSD.

Class	Minimum area
DeepGlobe, 0.5 m GSD
Urban area	0.1 ha
Vegetation ∗	0.125 ha
Water bodies	0.1 ha
Barren land	0.125 ha
EarthVQA, 0.3 m GSD

Appendix

Table 9: Classes in SQuID and their minimum area thresholds. Questions state these thresholds to reduce the effect of small segmentation artifacts.

Type	Question	GSD	Answer	Range
Tier 1: Basic Quantification (710 questions, 35.5%)
percentage	What percentage of the image is covered by barren land?	0.3 m	23.36%	[21.6, 25.1]
count	How many separate urban area regions are there? When counting, ignore patches smaller than 0.1 hectares.	0.5 m	4	[3, 5]
size	What percentage of the image is covered by the largest vegetation region, among regions larger than 0.125 hectares?	0.5 m	75.81%	[74.1, 77.6]
total_area	What is the total area of solar panels in hectares, excluding installations smaller than 0.01 hectares?	0.3 m	3.14 ha	[3.08, 3.2]
binary_comparison	Is there more barren land than forest area in this image?	0.3 m	yes	exact

Appendix

Table 10: Question examples from SQuID , one per type. The GSD is appended to each question in the dataset, and tolerance ranges follow Section 3 .

Question type	Count	Share	Description
Tier 1: Basic Quantification (710 questions, 35.5%)
count	178	8.9%	Number of regions above an area threshold
binary_comparison	172	8.6%	Whether one class covers more area than another
size	166	8.3%	Size of the largest or smallest region
percentage	157	7.8%	Percentage of the image covered by one class
binary_threshold	11	0.6%	Whether the total area exceeds a threshold

Appendix

Table 11: The 24 question types in SQuID.

Panel of Figure 7 — Figure 7: QASR developer prompt. The API descriptions are generated from the function docstrings and the topics from the segmentation models available on the server, so both can be extended without retraining.

Panel of Figure 8 — Figure 8: Further QASR program examples for a percentage, a counting and a proximity question. Section 4 shows a multi-condition example and Figure 9 a full pipeline.

Panel of Figure 9 — Figure 9: Full QASR pipeline example (GPT-5 + ConvNeXt). Left, the image and the predicted masks. Right, the dataset entry, the generated program and its output. The answer 7 falls within the tolerance range [4, 8] of the reference answer 6.

Panel of Figure 10 — Figure 10: gpt-oss-120B program examples. Left, a percentage prediction within the tolerance range. Right, a count of 9, outside the reference range [4, 6].

		QASR, predicted masks						End-to-end VLMs
Question type	N	QASR-A	QASR-B	QASR-C	QASR-D	QASR-E	QASR-F	GPT-5	Qwen3-VL-30B-A3B	Qwen3-VL-8B	GeoChat	Geo-R1	Qwen2.5-VL-7B
Tier 1: Basic Quantification
count	178	56.7	36.0	55.1	36.5	27.5	18.5	36.5	45.5	15.7	27.0	42.1	3.4
binary_comparison	172	78.5	76.7	79.7	57.6	46.5	61.6	67.4	78.5	75.0	100	52.9	57.0
size	166	33.7	23.5	33.7	24.1	4.2	19.9	16.3	12.7	3.0	4.8	10.8	11.4
percentage	157	36.9	33.1	26.1	31.2	2.5	5.1	26.8	11.5	7.0	2.5	6.4	7.6

Appendix

Table 12: Accuracy (%) by question type for 12 configurations. QASR-A to QASR-F are the configurations of Table 3 , run with predicted masks. Rows follow Table 11 , and the best value in each row is in bold.

Panel of Figure 11 — Figure 11: Qwen3-VL-30B-A3B-Thinking examples. Left, a proximity estimate within the tolerance range. Right, an incorrect area estimate. The displayed traces are shortened.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments
Sep 26, 2026Dongdong Wang, Qingqi Song, Yuzhou Chen +3VLM Reasoning Remote Sensing Benchmarks
University of Florida
SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models
Aug 3, 2026Hai Nguyen, Tung Vu, Cong TranSpatial Reasoning Benchmarks 3D Spatial Reasoning
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam
EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery
Feb 17, 2026Zelin Xu, Yupu Zhang, Saugat Adhikari +6Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA · Department of Computer Science, Indiana University Bloomington, Bloomington, IN, USA.

Sep 26, 2026J/K move · Enter open · S save

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

Dongdong Wang, Qingqi Song, Yuzhou Chen, Deepak Balakrishnan+2

University of Florida

VLM Reasoning Remote Sensing Benchmarks Remote Sensing VQA Visual Question Answering

Open paper AskPDF arXiv

Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery. Existing benchmarks and algorithms are predominantly based on qualitative Visual Question Answering (VQA), providing limited insights into the quantitative reasoning capabilities of VLMs for built environment metrics. To address this gap, we develop Quantitative Urban and Spatial AI benchmark (USAI-Quant), the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery. USAI-Quant is curated from the 335 largest U.S. cities, aligning high-resolution remote sensing images with quantitative built environment metrics. We then evaluate both general-purpose and remote sensing VLMs (RS-VLMs) by applying VQAs to tens of built environment metrics across three complexity levels. Our results reveal that current state-of-the-art models consistently fall short on numeric reasoning tasks. We further conduct in-depth analyses across models, question types, and geographic locations, uncovering insights into performance variability and task-specific challenges.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models

EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery