USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

Dongdong Wang; Qingqi Song; Yuzhou Chen; Deepak Balakrishnan; Ravi Shankar Srinivasan; Shenhao Wang

cs.CVSep 26, 2026

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

Authors: Dongdong Wang, Qingqi Song, Yuzhou Chen, Deepak Balakrishnan, Ravi Shankar Srinivasan, Shenhao Wang

Organizations: University of Florida

Abstract

Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery. Existing benchmarks and algorithms are predominantly based on qualitative Visual Question Answering (VQA), providing limited insights into the quantitative reasoning capabilities of VLMs for built environment metrics. To address this gap, we develop Quantitative Urban and Spatial AI benchmark (USAI-Quant), the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery. USAI-Quant is curated from the 335 largest U.S. cities, aligning high-resolution remote sensing images with quantitative built environment metrics. We then evaluate both general-purpose and remote sensing VLMs (RS-VLMs) by applying VQAs to tens of built environment metrics across three complexity levels. Our results reveal that current state-of-the-art models consistently fall short on numeric reasoning tasks. We further conduct in-depth analyses across models, question types, and geographic locations, uncovering insights into performance variability and task-specific challenges.

Figures & tables

Figure 1: An example from USAI-Quant. USAI-Quant is designed to evaluate quantitative reasoning for built environment metrics through remote sensing imagery. It integrates 13 urban metrics and supports both open-ended and multiple-choice question formats, enabling comprehensive assessment of quantitative VQAs.

Figure 2: Benchmark preparation pipeline.

Figure 3: Geographic locations of the 335 selected metropolitan areas used for urban imagery analysis.

Panel of Table 1 — Table 1: Comparing state‑of‑the‑art remote sensing VQA benchmarks with respect to both general performance criteria and urban‑focused evaluation metrics. ✗ indicates that no questions correspond to this feature, ✓ indicates that some questions are partially relevant to this feature, and ✔ indicates that all questions focus on this feature.

Metric	Formula	Description	Level
Building Footprint Ratio	$\sum A_{i}^{\text{footprint}}/A_{\text{total}}$	Share of land covered by buildings; development intensity.	L1
Commercial Area Percentage	$A_{\text{commercial}}/A_{\text{total}}$	Proportion of commercial land.	L1
Green Area Percentage	$A_{\text{green}}/A_{\text{total}}$	Vegetated land share; ecological capacity.	L1
Residential Area Percentage	$A_{\text{residential}}/A_{\text{total}}$	Housing land proportion.	L1
Urban Blue Area Percentage	$A_{\text{blue}}/A_{\text{total}}$	Surface water proportion.	L1
Industrial Area Percentage	$A_{\text{industrial}}/A_{\text{total}}$	Industrial land share.	L1

Table 2: Selected built environment metrics [ 34 , 28 , 13 ] with definitions, explanation and reasoning levels.

#	Prompt Template
1	What is the [XXX] in this area?
2	Can you describe the [XXX] at this location?
3	Please indicate the [XXX] for this location.
4	Could you provide the [XXX] value observed here?
5	Identify the [XXX] at this location.
6	Determine the [XXX] across this region.

Table 3: Alternative prompt templates using [XXX] as a placeholder for built environment metrics.

Metric	Value	Metric	Value
Total Images	100,523	Training Images	95,523
Total VQA Pairs	699,341	Training VQA Pairs	569,341
Metrics per Image	13	Testing Images	5,000
Question Length (words)	9.23	Testing VQA Pairs	130,000

Table 4: Overall statistics of the USAI-Quant.

Figure 4: Different built environment metric categories.

Built Environment Metrics	Level	General-Purpose Models				Domain-Specific Models
Built Environment Metrics	Level	LLaVA-1.6	InternVL-3	QwenVL-3	MGM	GeoChat	VHM	Falcon
Building Footprint Ratio	L1	78.5 / 5.1	49.4 / 4.1	41.9 / 2.1	47.9 / 5.9	97.8 / 12.7	93.5 / 8.6	93.8 / 10.2
Commercial Area Percentage	L1	79.3 / 8.4	57.5 / 4.1	62.6 / 3.0	77.1 / 4.2	95.9 / 10.2	92.9 / 12.2	92.7 / 10.3
Green Area Percentage	L1	76.8 / 20.3	61.6 / 16.3	59.6 / 14.7	83.9 / 17.1	90.9 / 35.8	97.8 / 30.7	92.1 / 29.5
Residential Area Percentage	L1	78.6 / 9.3	58.6 / 5.2	57.2 / 2.9	76.4 / 6.1	94.5 / 12.4	90.7 / 9.7	91.5 / 9.6
Urban Blue Area Percentage	L1	82.6 / 15.4	60.9 / 14.3	57.7 / 8.0	81.2 / 13.7	94.8 / 18.8	95.5 / 17.1	90.9 / 20.4

Table 5: Quantitative reasoning performance across built environment metrics. Each cell reports success rate (%) / numeric tolerance accuracy (%). Metrics are sorted by reasoning level.

Built Environment Metrics	Level	General-Purpose Models				Domain-Specific Models
Built Environment Metrics	Level	LLaVA-1.6	InternVL-3	QwenVL-3	MGM	GeoChat	VHM	Falcon
Building Footprint Ratio	L1	27.71	36.21	35.21	33.41	22.14	23.09	25.55
Commercial Area Percentage	L1	27.19	37.05	35.94	32.87	22.48	23.62	26.21
Green Area Percentage	L1	35.06	45.72	44.18	43.96	41.29	42.97	42.01
Residential Area Percentage	L1	25.84	36.90	35.63	34.25	22.71	23.02	24.89
Urban Blue Area Percentage	L1	24.97	34.88	33.94	32.36	21.85	23.66	26.02

Table 6: Comparison of quantitative reasoning accuracy (%) across different state-of-the-art large VLMs in multiple-choice question.

Figure 5: Accuracy of Qwen3-VL-8B across 10 templates.

Figure 6: Accuracy of Qwen3-VL-8B across cities.

Figure 7: Effect of model size on Qwen3-VL and InternVL-3.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery
Jan 19, 2026Peter A. Massih, Eric Cosatto, Martin Renqiang MinSpatial Reasoning Benchmarks Geospatial Reasoning Benchmarks
Department of Machine Learning, NEC Laboratories America · EPFL
Built Environment Reasoning from Remote Sensing Imagery Using Large Vision--Language Models
May 8, 2026Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan +1Remote Sensing Image Understanding Large Vision-Language Models
Department of Urban and Regional Planning, University of Florida · M.E. Rinker, Sr. School of Construction Management, University of Florida
SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
Feb 24, 2026Yuechen Xie, Xiaoyan Zhang, Yicheng Shan +6VLM Reasoning Spatial Reasoning Benchmarks
Zhejiang University · The University of Sydney · ManyCore +2

Jan 19, 2026J/K move · Enter open · S save

SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery

Peter A. Massih, Eric Cosatto, Martin Renqiang Min

Department of Machine Learning, NEC Laboratories America · EPFL

Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Remote Sensing Benchmarks Remote Sensing VQA Remote Sensing

Open paper AskPDF arXiv

Vision-Language Models (VLMs) describe satellite images fluently but struggle with the quantitative spatial reasoning that remote sensing (RS) analysts perform routinely, such as counting buildings, measuring forest area in hectares and computing distances in meters. We introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 questions over 587 images from four sources at 0.3 to 1.0 m ground sampling distance, covering 24 question types in three difficulty tiers. Answers are computed from curated segmentation masks or, for 50 questions, from human consensus, and numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers, so that modular systems can be evaluated using reference rather than predicted masks, and 75,687 training question-answer pairs on disjoint images (SQuID-Train). Among 15 evaluated systems, the best zero-shot end-to-end VLM reaches 28.1%, and RS-specialized VLMs score lower. As a baseline, we propose QASR (Quantitative Agentic Spatial Reasoning), in which an LLM writes a program that segments the image and measures the masks with geometric operators. QASR reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks instead of predicted masks raises accuracy from 42.2% to 82.6%, so segmentation is its main source of error. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%, close to QASR. Multi-condition questions (Tier 3) remain the hardest in every setting, at 62.0% for QASR with reference masks and 13.2% after fine-tuning. These results leave clear room for end-to-end models that learn metric spatial reasoning, and SQuID provides the data to train and evaluate them.

Abstract

VLM Reasoning

Explore similar work

SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery

Built Environment Reasoning from Remote Sensing Imagery Using Large Vision--Language Models

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models