DisasterVLG: Vision-Language Grounding and Evidence-Based Reasoning for Disaster Scene Understanding
Organizations: Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar · College of Science & Engineering, Hamad Bin Khalifa University, Doha, Qatar
Abstract
General-purpose vision-language models struggle with crisis imagery due to its inherent noise, ambiguity, and high variability across disaster events - yet timely, accurate scene understanding is critical for humanitarian response. To address this, we introduce DisasterVLG, a large-scale multi-task dataset containing 1.24 million disaster images from 142 countries paired with 33 million structured annotations spanning dense captioning with localization, incident assessment, relational scene descriptions, and crisis-grounded visual question answering (VQA) across diverse hazard types and scene contexts worldwide. Unlike prior disaster benchmarks that focus narrowly on classification, DisasterVLG unifies recognition, grounded reasoning, and uncertainty-aware interpretation. A distinguishing feature is its evidence-grounded annotation design: each VQA instance includes supporting evidence text that links visual observations to answers, alongside positive and negative question pairs, multiple answer formats, and difficulty labels. Visual grounding layer links caption phrases to image regions and relationship statements to subject-object boxes, with explicit labels for visual supports and clues. To support rigorous evaluation, we construct a human-verified test set of 38K images (1.15M annotations) spanning broad geographic regions and disaster types. Experiments with domain-adapted vision-language models demonstrate strong performance across disaster captioning, incident recognition, and grounded VQA, while also exposing the limitations of general-purpose models in crisis-specific reasoning. DisasterVLG establishes a scalable, reproducible benchmark for vision-language models that must reason reliably under real-world uncertainty. We will publicly release the dataset, trained models, and code.
Figures & tables
| View | Dataset | Images | Ann. | Classification | Segmentation | Captioning | VQA | MCQ | Visual Grounding | Reasoning |
| Aerial | Sen1Floods11 [ 12 ] | 4,831 | - | ✓ | ||||||
| Sen12Landslides [ 25 ] | 39K | - | ✓ | |||||||
| xBD [ 22 ] | 22K | - | ✓ | |||||||
| FloodNet [ 39 ] | 2,348 | 7,345 | ✓ | ✓ | ✓ | |||||
| DisasterM3 [ 45 ] | 27K | 123K | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Ground | MEDIC [ 7 ] | 72K | - | ✓ |
| Scale | Annotation & VQA Structure | ||||||||||||
| Split | Images | CT | IT | QA | Q/I | MCQ | VG | Rel/Unc/Loc | CapLen | Pos/Neg | FF | E/M/H | Supp/Abs |
| Train | #1,197,351 | 29 | 12K | 19.9M | 16.64 | 2.6M | 7.59M | 97.5 / 98.0 / 69.04 | 88.78 | 79.8 / 20.2 | 59.6 | 57.6 / 38.4 / 4.0 | 1.56 / 0.34 |
| Test | 38,254 | 28 | 1,498 | 641K | 16.76 | 85K | 350K | 99.0 / 99.7 / 59.70 | 83.78 | 79.1 / 20.9 | 62.7 | 51.9 / 42.4 / 5.8 | 1.55 / 0.35 |
| Full | 1.24M | 29 | 12K | 20.5M | 16.65 | 2.7M | 7.89M | 97.5 / 98.1 / 69.12 | 88.63 | 79.8 / 20.2 | 59.7 | 57.4 / 38.5 / 4.1 | 1.56 / 0.34 |
| Captioning | Visual Grounding | VQA Reasoning | Incident Reasoning | MCQ | |||||||||
| Model | R-L | ST5 | GES | D-F1 | GC-F1 | R-L | ST5 | GES | R-L | ST5 | GES | Acc | Acc |
| Molmo2 [ 15 ] | 23.89 | 81.90 | 52.4 | 46.4 | 39.6 | 17.96 | 75.24 | 53.9 | 21.25 | 73.64 | 55.2 | 70.84 | 63.57 |
| InternVL3.5-8B [ 47 ] | 25.12 | 83.45 | 56.7 | 53.6 | 47.5 | 19.09 | 77.89 | 56.8 | 22.87 | 76.59 | 56.9 | 72.60 | 65.83 |
| Qwen3-VL-4B [ 10 ] | 23.98 | 83.20 | 54.5 | 47.2 | 42.5 | 17.80 | 75.83 | 54.1 | 19.78 | 75.59 | 55.6 | 69.92 | 62.44 |
| Qwen3-VL-8B [ 10 ] | 24.29 | 83.15 | 56.2 | 52.5 | 45.1 | 18.73 | 77.25 | 53.6 | 19.16 | 73.78 | 55.7 | 70.61 | 63.14 |
| Qwen3.5-2B [ 44 ] | 24.26 | 81.40 | 49.8 | 43.5 | 37.9 | 17.61 | 75.78 | 51.7 | 18.05 | 72.22 | 53.5 | 68.85 | 61.73 |
| Model | Object Macro-F1 | Scene Macro-F1 |
| Qwen3.5-9B | 73.6 | 76.1 |
| DisasterVLM-4B | 77.9 | 78.2 |
| DisasterVLM-9B | 81.5 | 82.85 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Full dataset | Train set | Test set |
| Total images | 1,235,605 | 1,197,351 | 38,254 |
| Canonical categories | 29 | 29 | 28 |
| Total QA pairs | 20,568,496 | 19,927,329 | 641,167 |
| Average QA pairs / image | 16.65 | 16.64 | 16.76 |
| Median QA pairs / image | 17 | 17 | 17 |
| Total Visual Grounding Boxes | 7,948,308 | 7,597,459 | 350,849 |
Vision-Language Grounding
Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.
See all 434 papers