DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Response

Sara Tehrani; Yonghao Xu; Leif Haglund; Amanda Berg; Gulnaz Zhambulova; Michael Felsberg

cs.CVJan 26, 2026

DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Response

Authors: Sara Tehrani, Yonghao Xu, Leif Haglund, Amanda Berg, Gulnaz Zhambulova, Michael Felsberg

Organizations: Computer Vision and Learning System, Linköping University, Sweden · Vantor, Linköping, Sweden

Abstract

Vision--language models (VLMs) show promise for disaster-response remote sensing, but existing benchmarks mainly emphasize scene-level or damage-centric assessment. To study this building-centric gap, we introduce \method{}, a diagnostic benchmark built on xBD, a pre/post-disaster satellite dataset with building-level damage labels. \method{} enriches building instances with OpenStreetMap-derived functional labels and contains 134{,}108 task-specific instruction records across 15 task types, spanning instance-level assessment, scene-level counting, multi-instance reasoning, and structured report generation. The benchmark supports RGB pre/post-disaster imagery, single- and multi-view instance formulations, and scene-level RGB/SAR diagnostic inputs. Experiments with general-domain and remote-sensing VLMs show that models perform better on visible damage cues than on building-function understanding, multi-instance reasoning, counting, and grounded reporting. Instruction tuning improves performance on several tasks but does not close this building-centric gap.

Figures & tables

Figure 1 : Scene-level summaries can miss building-level evidence. DisasterInsight grounds disaster assessment at the building-instance level.

Dataset	Bldg. Inst.	Damage	Modality	Report	Function
RSICD [ 15 ]	✗	✗	Optical + caption	Caption	✗
BRIGHT [ 5 ]	✗	✓	Optical/SAR	✗	✗
FloodNet [ 20 ]	✗	✓	UAV RGB + QA	VQA	✗
xBD [ 8 ]	✓	✓	Pre/post optical	✗	✗
CrisisMMD [ 1 ]	✗	Partial	Image + text	Summ.	✗
CrisisFACTS [ 16 ]	✗	✗	Text	Summ.	✗

Table 1 : Comparison with representative disaster and vision–language datasets. “Bldg. Inst.” indicates building-instance grounding, and “Function” indicates explicit building-use labels.

Figure 2 : Overview of DisasterInsight: OSM-enriched xBD buildings and tasks for instance, scene, multi-instance, and report generation. © ESA/Sentinel.

Figure 3 : OSM matching examples with accepted aligned footprints and rejected distant or missing matches. © OpenStreetMap contributors.

Family	Task types	Input	Metrics
Instance	Building-function classification; damage-level estimation; temporal-change detection; joint function–damage assessment	Target building views with pre/post imagery	Acc., macro-F1, MAE, exact match
Scene	Total building count; damage-level count; function count	Full pre/post scene imagery	MAE, RMSE, key-wise MAE
Report	Short structured report generation; long structured report generation	Full scene imagery; RGB–SAR scene setting	BLEU-4, ROUGE-L, BERTScore
Multi-instance	Selection by damage; selection by function; joint function–damage selection; damage comparison; spatial reasoning; response-priority reasoning	Marked buildings in pre/post scene imagery	Acc., exact match, Set-F1

Table 2 : DisasterInsight task taxonomy. The benchmark contains 15 task types grouped by evaluation family.

Figure 4 : Examples of DisasterInsight task conversations.

Figure 5 : Benchmark statistics across task families, disaster types, building-function labels, and damage categories.

Open-source models
	Instance-level						Multi-instance
	Func.		Dmg.		Temp.		F-Sel.	Spatial	Priority
Model	Acc. $\uparrow$	mF1 $\uparrow$	Acc. $\uparrow$	MAE $\downarrow$	Acc. $\uparrow$	mF1 $\uparrow$	Set-F1 $\uparrow$	Acc. $\uparrow$	Acc. $\uparrow$
Qwen2.5-VL-7B	74.9	21.0	81.2	0.32	44.8	16.8	66.5	47.4	33.3
Qwen3-VL-8B	86.6	25.9	72.7	0.46	44.1	27.9	66.4	53.4	31.0
Qwen3-VL-30B	91.0	30.1	64.2	0.69	22.7	13.3	70.3	39.8	31.0

Table 3: Instance- and multi-instance results on the standardized DisasterInsight test split. Func., Dmg., and Temp. denote building function, damage level, and temporal change. F-Sel., Spatial, and Priority denote multi-instance function selection, spatial reasoning, and response-priority reasoning. All values are percentages except Dmg. MAE.

Open-source models
	Counting			Reports
Model	Tot. $\downarrow$	Func. $\downarrow$	Dmg. $\downarrow$	B4 $\uparrow$	R-L $\uparrow$	BS $\uparrow$
Qwen2.5-VL-7B	66.0	6.5	19.2	1.8	16.3	84.3
Qwen3-VL-8B	175.6	9.8	20.5	3.1	17.7	84.8
Qwen3-VL-30B	69.7	15.2	18.3	2.2	16.6	84.2
LLaVA-OneVision	78.6	5.7	20.2	3.4	19.4	85.4

Table 4: Scene-level counting and report-generation results. Counting uses total-count MAE and key-wise MAE for function and damage counts; report metrics are BLEU-4, ROUGE-L F1, and BERTScore F1 percentages.

Figure 6 : Qualitative examples across task families.

Figure 7 : Gap-analysis summary.

OSM Labels	Benchmark Category
house , apartments , residential , detached , bungalow , static_caravan	residential
hospital , clinic , pharmacy , doctors , dentist , veterinary	medical_facilities
school , college , university , kindergarten , childcare	educational_facilities
fire_station , police , courthouse , shelter , prison , townhall , government	government_emergency
supermarket , convenience , mall , restaurant , fast_food , bank , retail , commercial , doityourself , computer , car_wash , pet	commercial
warehouse , factory , storage_rental , industrial , fuel , power , works	industrial_utilities

Table A.1 : Representative mapping from normalized OSM semantic labels to benchmark building-use categories.

Panel of Figure 13 — Figure A.1 : OSM enrichment filtering statistics for xBD buildings. The final function-tagged subset is intentionally conservative, reflecting sparse and noisy OSM coverage.

Figure A.2 : Additional representative examples from DisasterInsight.

Explore similar work

Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagery
Aug 3, 2026Huy Quang Ung, Guillaume Habault, Roberto Legaspi +3Post-Disaster Damage Assessment Structural Damage
KDDI Research, Inc., Fujimino, Japan
GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence
Jun 15, 2026Maram Hasan, Aman Verma, Savitra Roy +5Geospatial Reasoning Disaster Response
Indian Institute of Technology Bombay · Mohamed bin Zayed University of Artificial Intelligence
Damage-TriageFormer: A Foundation-Model Framework for Typology-Based Building Damage Assessment from Mono-Temporal Imagery
Jun 10, 2026Yiming Xiao, Yu-Hsuan Ho, Sanjay Thasma +2Post-Disaster Damage Assessment Structural Damage
Texas A&M University · Resilitix Intelligence LLC · Institute for a Disaster Resilient Texas

Aug 3, 2026, cs.CVJ/K move · Enter open · S save

Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagery

Huy Quang Ung, Guillaume Habault, Roberto Legaspi, Hao Niu+2

KDDI Research, Inc., Fujimino, Japan

Post-Disaster Damage Assessment Structural Damage Aerial Imagery Object Detection Deep Learning Unmanned Aerial Vehicles

Open paper AskPDF arXiv

Rapid and accurate post-disaster building damage assessment is essential, yet remains a challenging task. Unmanned Aerial Vehicle (UAV) imagery offers a timely and high-resolution view of affected areas, but existing Computer Vision (CV) models often demand large annotated datasets, generalize poorly across geographic regions and their assessment policies, and are confined to the specific tasks they were trained for. Large Vision-Language Models (LVLMs) offer a promising alternative through their strong reasoning and generalization capabilities, but fall short on precise, low-level perception tasks such as object detection and accurate bounding box generation. Furthermore, they often require a substantial amount of data for effective fine-tuning on domain-specific tasks. In this paper, we propose a hybrid framework that decouples detection from damage assessment, combining the precision of CV models with the reasoning power of LVLMs. A CV model first detects buildings and generates bounding boxes on the image that are then passed to an LVLM for damage classification and contextual interpretation. We evaluated our framework on two real-world benchmarks: RescueNet and FloodNet. In particular, the best combination under this framework accurately counts intact, partially damaged and completely destroyed buildings, surpassing isolated baselines by up to 2.1 R^2 points, while requiring only limited annotated data for the detection stage. Beyond reporting aggregate gains, we provide a detailed analysis of failure scenarios and edge cases, offering practical insights for practitioners and concrete directions for future work. Our source code and data are publicly available to the research community via the following repository: https://github.com/ungquanghuy-kddi/VLM_GDINO.git

Abstract

Explore similar work

Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagery

GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence

Damage-TriageFormer: A Foundation-Model Framework for Typology-Based Building Damage Assessment from Mono-Temporal Imagery