A single unmanned aerial vehicle (UAV) mission over an incident site produces hundreds of images, far more than a rescue commander can review under time pressure, and human screening of destruction imagery carries a documented psychological cost. The two obvious remedies both fail in search and rescue (SAR) practice: cloud vision-language model (VLM) services are excluded because the imagery must not leave the incident network under GDPR and German civil-protection (BOS) rules, and fine-tuned specialist detectors age quickly while retraining and re-certifying them exceeds most SAR organizations resources. We present RescueVLM, an open-source, training-free and model-agnostic triage tool built on SketchVLM: a general-purpose open-weight VLM runs fully on-premise and is swapped for a newer model through a simple dropdown selection, so triage quality grows with the underlying models because all task knowledge lives in prompts and an output contract rather than in trained weights. One natural-language query over a mission folder returns a ranked shortlist with per-image justifications, editable SVG overlay annotations, and a GPS map of the hits. In a qualitative evaluation on German fire-brigade field data from three real deployments and a live fire exercise, a single query condenses an entire survey mission into a handful of annotated frames that a commander can review at a glance.
Figures & tables
Fig. 1: RescueVLM triage run on the flood use case, Erftstadt/Blessem 2021 (fast mode): image grid with ranked hits, EXIF-GPS map, and situation summary. The hits flag severe flood- and erosion-related hazards: nearly submerged, likely structurally compromised buildings, a collapsed riverbank forming a dangerous precipice, deep gullies in agricultural fields posing fall and vehicle-entrapment risk, flood debris and mud encroaching on a residential area, and houses at the edge of the flood path where water is undermining the surrounding ground.
Use case
Incident
Pattern
Images
ABC reconnaissance
Industrial hall, Berlin 2021
meander
167
Flood
Erftstadt/Blessem 2021
meander
140
Structural fire
Apartment building, Essen 2022
meander
159
Vegetation fire
Exercise, Viersen 2025
meander
52
TABLE I: Field datasets and the SAR use case each exercises.
Fig. 2: Four of the shortlisted frames for the ABC-reconnaissance use case (burned-out industrial hall, Berlin), with the model’s rank, annotation, and ∼ 10-word justification: (1) major roof collapse, (2) personnel at risk, (3) traffic beside the unstable structure, (5) the spill flagged as a potential chemical hazard.
Fig. 3: Four of the shortlisted frames for the flood use case (Erftstadt/Blessem), with the model’s rank, annotation, and ∼ 10-word justification: (1) vehicle nearly surrounded by floodwater, (2) deep erosion pits and unstable land, (3) fragmented land segments prone to collapse, (4) floodwater encroaching on buildings.
Fig. 4: Four of the shortlisted frames for the structural-fire use case (Essen), with the model’s rank, annotation, and ∼ 10-word justification: (1) severe structural damage near a public path, (2) active emergency response scene, (3) people near the unstable structure, (4) emergency-vehicle congestion.
Fig. 5: Hazard prioritization by gemma4:31b in RescueVLM on the Viersen dataset. Right: ranked summary of the most critical images, distinguishing risk to infrastructure, risk to personnel, and environmental hazards, with each finding referenced by image index and file name. Left: the top-ranked result (Image 30), in which the model marks the active fire and the adjacent residential building. The annotations are grounded in the image content and directly linked to the textual assessment.
Fig. 6: Hallucinations by nemotron3:33b . Left: the model draws a rectangle and labels it as people, although the marked region contains no people and lies away from the center of the scenario. Right: the model places a circle at an arbitrary location and describes it as structural damage and a fire truck ladder, although there is no damage at that position and no ladder in the image. In both cases, the annotations are confidently labeled but not grounded in the image content.
Vision--language models (VLMs) show promise for disaster-response remote sensing, but existing benchmarks mainly emphasize scene-level or damage-centric assessment. To study this building-centric gap, we introduce \method{}, a diagnostic benchmark built on xBD, a pre/post-disaster satellite dataset with building-level damage labels. \method{} enriches building instances with OpenStreetMap-derived functional labels and contains 134{,}108 task-specific instruction records across 15 task types, spanning instance-level assessment, scene-level counting, multi-instance reasoning, and structured report generation. The benchmark supports RGB pre/post-disaster imagery, single- and multi-view instance formulations, and scene-level RGB/SAR diagnostic inputs. Experiments with general-domain and remote-sensing VLMs show that models perform better on visible damage cues than on building-function understanding, multi-instance reasoning, counting, and grounded reporting. Instruction tuning improves performance on several tasks but does not close this building-centric gap.
Sara Tehrani, Yonghao Xu, Leif Haglund +3
Computer Vision and Learning System, Linköping University, Sweden · Vantor, Linköping, Sweden
Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs.\ high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes
We show that an appropriate visual harness can substantially improve the performance of general vision-language models (VLMs) across diverse medical imaging tasks. We introduce VERA, a medical visual harness that equips general VLMs with interactive visual tools and systematic diagnostic evidence management. With VERA, general VLMs interactively explore medical images and progressively acquire diagnostic evidence, which is preserved in a lossless memory for subsequent retrieval and use. After submitting an initial answer, the VLM receives an assessment of potential omissions and the reliability of identified findings. This assessment guides further image inspection and measurement, followed by answer revision in an iterative agentic loop until sufficient evidence is obtained. Across 11 tasks and four imaging modalities, VERA improves performance on all tasks with Qwen3.8-27B, with notable gains in spleen measurement (63.3% to 98.3%), renal search (57.5% to 71.2%), and pulmonary nodule detection (0.0% to 28.2%). High-capability API-based VLMs achieve even higher performance with VERA. For example, GPT-6.1 Sol improves pulmonary nodule detection from 10.8% to 69.2% with VERA, demonstrating the complementary benefits of stronger model capabilities and an effective medical visual harness. Our results highlight the potential of visual harnesses as a new paradigm for scaling general VLMs to medical imaging tasks.
Shaowen Wan, Hanqi Jiang, Zihao Wu +6
New Jersey Institute of Technology · University of Georgia · New York University