A Simulation-Grounded Agentic VLM Framework for Wildfire Monitoring and Reporting

Duowen Chen; Yuchen Sun; Zhiqi Li; Yuxuan Liao; Sinan Wang; Bart van Bloemen Waanders; Bo Zhu

cs.CVOct 1, 2026

A Simulation-Grounded Agentic VLM Framework for Wildfire Monitoring and Reporting

Authors: Duowen Chen, Yuchen Sun, Zhiqi Li, Yuxuan Liao, Sinan Wang, Bart van Bloemen Waanders, Bo Zhu

Organizations: Georgia Institute of Technology · Sandia National Laboratories

Abstract

Effective wildfire monitoring requires relating visual evidence to physical fire dynamics, yet real videos with synchronized physical annotations are scarce and high-fidelity 3D simulation is costly. We present a simulation-grounded vision-language model (VLM) framework that automatically converts 2D wildfire simulations into labeled video episodes. A fixed Blender mapping produces low-detail 3D proxies aligned with simulator terrain, fuel layout, fire activity, and wind cues; controllable video generation supplies richer appearance. The proxies are intermediate representations rather than finely rendered final scenes. Generated videos and simulator labels form reusable multimodal memory for a training-free multi-agent VLM system that retrieves reference episodes, reconciles visual and memory-based predictions, and produces structured wildfire reports. On held-out generated episodes, video memory achieves 51.5% exact four-tag accuracy, compared with 22.6% for direct VLM querying and 16-17% for text-only memory; the complete system achieves 77.3% accuracy on six simulator-derived report fields. Component ablations, cross-generator tests, and three real-UAV evaluations assess retrieval, reporting, generator changes, and observable monitoring tasks. The framework connects automatic simulation-to-proxy conversion with memory-based VLM reasoning under scarce real-world physical annotations.

Figures & tables

Figure 1: Overall simulation-grounded VLM framework. Top: 2D simulation and procedural 3D proxies provide depth, scribble, and prompt controls for synthetic aerial-view episodes with simulator-derived labels. Bottom: a memory-augmented multi-agent VLM reporter combines query frames, retrieved labeled references, and disagreement adjudication for wildfire monitoring and reporting.

Figure 2: Simulation-grounded synthetic wildfire episodes. Left: SimFire propagation under different settings and locations. Middle: procedural 3D billboard rendering from an aerial viewpoint. Right: generated videos conditioned on the proxy renderings. Episode labels come from the originating simulations.

Figure 3: Agentic VLM reporting pipeline. A query wildfire video is analyzed using direct visual evidence and retrieved physics-labeled memory evidence. Agreement across reporters is written to the final report, while disagreement is routed to a judge. The resolved predictions are rendered as a structured HTML report.

Figure 5: We show generated video frames with different simulation tags (2 video clips for each tag).

Figure 6: Illustrative reports show predictions, visual evidence, location, and response suggestions; these outputs are not operationally validated.

Task	Direct Query GPT	Multi-Agent + Combined Memory	ForestFireVLM	Qwen2-Wildfire-2B
Fast Spread Risk	55.7%	75.3%	58.8%	43.3%
High Peak Burning Area	63.9%	77.3%	54.6%	59.8%
High Final Burnscar	64.9%	80.4%	55.7%	60.8%
High Canopy Cover	57.7%	73.2%	67.0%	47.4%
Rapid Growth + Large Burn	64.9%	79.4%	55.7%	60.8%
Severe Fire Behavior	66.0%	78.4%	56.7%	59.8%

Table 4: Accuracy on six simulator-derived report fields for held-out generated episodes. Higher is better. Bold and underlined values mark the best and second-best results, respectively.

(a) Cross-generator evaluation
Query $\rightarrow$ Memory	Ours	GPT	Claude	Gemini
LTX $\rightarrow$ VACE	41.00	19.00	24.00	28.00
VACE $\rightarrow$ LTX	44.44	27.27	23.23	26.26

Table 5: Generalization tests. (a) Cross-generator accuracy (%) on generated episodes; LTX denotes LTX-2.3 + AVControl. (b) Real-UAV smoke/fire detection and visible fire-area assessment: Boreal accuracy is a percentage; other scores are on a 0–1 scale. Best baseline is selected separately for each metric, not one common model. Bold includes ties. Full real-UAV results appear in Appendix L .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Class balance	Majority	Random	Stratified
Wind	high: 49.5%, low: 50.5%	50.5%	50.0%	50.0%
Moisture	high: 49.5%, low: 50.5%	50.5%	50.0%	50.0%
Slope	high: 44.4%, low: 55.6%	55.6%	50.0%	50.6%
Vegetation	high: 47.5%, low: 52.5%	52.5%	50.0%	50.1%

Appendix

Table 10

Task	Class balance	Majority	Random	Stratified
Fast Spread Risk	fast: 56.7%, slow/moderate: 43.3%	56.7%	50.0%	50.9%
High Peak Burning Area	high: 40.2%, low: 59.8%	59.8%	50.0%	51.9%
High Final Burnscar	high: 39.2%, low: 60.8%	60.8%	50.0%	52.3%
High Canopy Cover	high: 44.3%, low: 55.7%	55.7%	50.0%	50.6%
Rapid Growth + Large Burn	high: 39.2%, low: 60.8%	60.8%	50.0%	52.3%
Severe Fire Behavior	high: 40.2%, low: 59.8%	59.8%	50.0%	51.9%

Appendix

Table 11

Task	Meaning	Positive Label Definition
Fast Spread Risk	The fire spreads at a fast or very fast rate.	Positive if $v_{\max}\geq 0.30$ m/s. We use four spread classes: slow for $v_{\max}<0.15$ , moderate for $0.15\leq v_{\max}<0.30$ , fast for $0.30\leq v_{\max}<0.45$ , and very fast for $v_{\max}\geq 0.45$ .
High Peak Burning Area	The episode reaches a large active burning area.	Positive if $A_{\mathrm{peak}}\geq 90{,}000$ m 2 .
High Final Burn Scar	The episode leaves a large final burned area.	Positive if $A_{\mathrm{final}}\geq 300{,}000$ m 2 .
High Canopy Cover	The episode occurs in a relatively high-canopy environment.	Positive if $c$ is at or above the median canopy cover in the training set. In our data, this threshold is $2.61\%$ .
Rapid Growth + Large Burn	The episode combines rapid fire spread with a large final burn scar.	Positive if $v_{\max}\geq 0.30$ m/s and $A_{\mathrm{final}}\geq 300{,}000$ m 2 .
Severe Fire Behavior	The episode satisfies multiple indicators of severe fire behavior.	Positive if at least two of the following three conditions hold: $v_{\max}\geq 0.30$ m/s, $A_{\mathrm{peak}}\geq 90{,}000$ m 2 , and $A_{\mathrm{final}}\geq 300{,}000$ m 2 .

Appendix

Table 8: Binary simulator-derived report-field definitions. Here

v_{\max}

is the maximum radial spread rate,

A_{\mathrm{peak}}

is the peak burning area,

A_{\mathrm{final}}

is the final burn-scar area, and

c

is the mean canopy-cover percentage.

Code	Name	Base billboard mix in renderer
1	Short grass	grass_short 94%, shrub_green 6%
2	Timber (grass + understory)	grass_short 62%, shrub_green 26%, tree_conifer 12%
3	Tall grass	grass_tall 94%, tree_hardwood 6%
4	Chaparral	shrub_green 65%, shrub_dry 20%, grass_tall 15%
5	Brush	shrub_green 42%, shrub_dry 33%, grass_short 25%
6	Dormant brush / hardwood slash	shrub_dry 55%, grass_short 20%, slash 25%

Appendix

Table 13

Figure 9: Geographic locations for memory construction and held-out evaluation; no VLM weights are updated.

Metric	Directions	Meaning
SSIM	$\uparrow$	Grayscale structural similarity between aligned VACE and render frames.
Edge F1 / Edge-control F1	$\uparrow$	Canny-edge overlap between VACE and render frames with a small dilation tolerance.
RGB L1	$\downarrow$	Mean absolute RGB difference over aligned frames.
LPIPS	$\downarrow$	AlexNet LPIPS perceptual distance averaged over aligned frames.
Correct-pair win rate	$\uparrow$	Fraction of samples for which the paired render is closer to the VACE output than a mismatched render.

Appendix

Table 11: Proxy–generated alignment metrics.

SSIM	SSIM Win Rate	Edge F1	Edge F1 Win Rate	RGB L1	RGB L1 Win Rate	LPIPS	LPIPS Win Rate
0.334	76.6%	0.612	81.3%	0.189	72.0%	0.516	99.1%

Appendix

Table 12: Proxy–generated alignment and correct-pair win rates.

Metric	$\leq 30^{\circ}$	$\leq 45^{\circ}$	$\leq 60^{\circ}$	Median	Mean
Smoke plume axis	95.8%	97.4%	99.2%	3.7	7.6
Growth-front fire-spread axis	47.1%	67.3%	79.0%	32.7	35.8

Appendix

Table 18

Cue or comparison	Score
Fire-front location	79.17
Fuel distribution	68.75
Burned area	70.83
Terrain structure	81.25
Overall	75.00
Identical proxy–proxy pairs	$\approx 95$

Appendix

Table 14: VLM-assessed agreement on a 0–100 scale. Identity-pair values are approximate overall averages, not per-cue scores.

Model family	Evaluation role	Prompt family
GPT-4.1 / GPT-5.2	direct query-only four-tag inference	shared query-only tag prompt in Section G.1
Gemini 2.5 Pro	direct query-only four-tag inference	shared query-only prompt plus Gemini/Claude JSON wrapper in Section G.2
Claude Sonnet 4	direct query-only four-tag inference	shared query-only prompt plus Gemini/Claude JSON wrapper in Section G.2
Qwen2-Wildfire-2B	local VLM four-tag inference; six-task downstream conditioned on its own predicted tags	local VLM binary-tag prompt in Section G.3 ; downstream VLM prediction in Section I.7
ForestFireVLM-7B	local VLM four-tag inference; six-task downstream conditioned on its own predicted tags	local VLM binary-tag prompt in Section G.3 ; downstream VLM prediction in Section I.7
GPT-5-mini	downstream specialist inference	downstream specialist system + context blocks in Section I

Appendix

Table 20

Input	Decision rule	Prompt effect
Fuel percentage by code	The top four burnable land-cover groups are selected after grouping by semantic cover type.	Forms the phrase “Surface cover should read as …”
Non-burnable percentage by code	The top three non-burnable groups are selected after grouping by semantic barrier type.	Forms the phrase “with … acting as fire breaks.”
Cover percentage $p$	$p\geq 60$ : dominant; $30\leq p<60$ : major; $10\leq p<30$ : moderate; $3\leq p<10$ : minor; $p<3$ : trace.	Adds qualitative size words such as “major brushy shrubland” or “minor open water.”
Burnable percentage $b$	$b<60$ : fragmented; $60\leq b<85$ : partially interrupted; $b\geq 85$ : mostly continuous.	Sets the fuel-continuity phrase.
Non-burnable percentage $n$	$n\geq 25$ : substantial; $10\leq n<25$ : noticeable; $n<10$ : limited.	Sets the barrier-influence phrase.

Appendix

Table 21

Wind speed	Wind phrase	Smoke phrase	Spotting phrase
$<0.75$ m/s	very light	Mostly vertical smoke column with gentle drift and limited sideways shear.	Only sparse short-range ember lofting where continuity supports it.
$0.75$ – $<2.0$ m/s	light	Mostly upright smoke plume with mild downwind drift and restrained lateral shear.	Limited ember lofting and short-range spotting where continuity supports it.
$2.0$ – $<5.0$ m/s	moderate	Clear downwind smoke advection with moderate plume tilt and lateral drift.	Occasional ember spotting where continuity supports it.
$5.0$ – $<8.0$ m/s	strong	Strong downwind smoke transport with a clearly bent plume and sustained lateral advection.	Frequent downwind ember transport and intermittent spotting where continuity supports it.
$\geq 8.0$ m/s	very strong	Strong downwind smoke transport with a clearly bent plume and sustained lateral advection.	Frequent downwind ember transport and intermittent spotting where continuity supports it.

Appendix

Table 22

Feature	Decision rule	Prompt effect
Moisture tag	low_moisture : very dry surface fuels; high_moisture : comparatively moister surface fuels.	Tags override numeric moisture wording.
Numeric moisture $m$	$m<0.03$ : very dry; $0.03\leq m<0.06$ : dry; $0.06\leq m<0.10$ : moderately dry; $m\geq 0.10$ : moist.	Used when no explicit moisture tag is present.
Elevation relief $r$	$r<100$ m: gentle; $100\leq r<300$ m: rolling; $300\leq r<600$ m: rugged; $r\geq 600$ m: very rugged.	Sets the terrain phrase. Default wording says spread accelerates on favorable slopes and eases where aspect is less favorable.
Slope tag	high_slope : emphasizes steeper local slopes; low_slope : emphasizes broader gentler local slopes.	Modifies terrain wording beyond the numeric relief class.
Vegetation tag	high_vegetation : denser vegetation, continuous canopy, fewer gaps; low_vegetation : more open, sparse vegetation, patchier canopy.	Adds a separate vegetation-density sentence.

Appendix

Table 23

Condition	Additional negative phrases
$w<0.75$ m/s	strong wind; very strong wind; gale-force smoke; wind-whipped flames; violent lateral smoke; long horizontal smoke bands; storm-like plume bending.
$0.75\leq w<2.0$ m/s	strong wind; gale-force smoke; wind-whipped flames; extreme lateral smoke advection; fully horizontal smoke plume.
$2.0\leq w<5.0$ m/s	hurricane-force wind; perfectly still air; fully vertical plume with no drift.
$w\geq 5.0$ m/s	still air; calm smoke column; perfectly vertical plume with no downwind drift.
low_moisture	rain-soaked fuels; wet ground everywhere.
high_moisture	ultra-dry fuels; explosive firestorm behavior from uniformly desiccated vegetation.

Appendix

Table 24

Model	All four	At least three	At least two
GPT-4.1 mini	24.27	57.65	91.92
GPT-5 mini	27.27	66.67	95.96
Claude Sonnet 4.5	23.23	63.64	95.96
Gemini 2.5 Pro	26.26	75.76	97.98

Appendix

Table 20: Environmental driver prediction with 16 labeled reference videos per query. All entries are percentages.

Reranker	Execution	Mean	Median	Prep. + scorer	Four-tag accuracy
GPT-5	API	9.18	8.60	11.80	51.52
Claude Sonnet 4.5	API	12.30	11.97	14.59	46.46
Gemini 2.5 Pro	API	22.24	21.59	24.53	51.52
Qwen3-VL-Reranker-2B	Local H200	5.79	5.16	7.97	40.40

Appendix

Table 21: Reranker ablation. Latencies are in seconds and exact four-tag accuracy is a percentage. Mean and median refer to scorer latency in seconds.

$k$	Exact four-tag hit	Marginal gain	Wind	Moisture	Slope	Vegetation
1	51.52	–	74.75	83.84	82.83	89.90
2	57.58	+6.06	79.80	88.89	88.89	91.92
3	64.65	+7.07	81.82	94.95	92.93	92.93
4	69.70	+5.05	87.88	95.96	95.96	97.98

Appendix

Table 22: Retrieval-size ablation. Hit rates are percentages and marginal gains are percentage-point increases in exact four-tag hit rate. These are retrieval metrics, not downstream prediction accuracies.

Method	Accuracy	Precision	Recall	F1
Ours	100.0	100.0	100.0	100.0
GPT-5	94.4	90.0	100.0	94.7
Claude 4	100.0	100.0	100.0	100.0
ForestFireVLM-7B	100.0	100.0	100.0	100.0
Qwen2-Wildfire-2B	94.4	100.0	88.9	94.1

Appendix

Table 23: Boreal Forest Fire smoke detection. All entries are percentages.

Method	Balanced accuracy	Accuracy
GPT-5 mini	0.7646	0.8173
Claude Sonnet 4.6	0.8563	0.9135
Gemini 2.5 Pro	0.8125	0.9135
ForestFireVLM-7B	0.8625	0.9231
Qwen2-Wildfire-2B	0.8375	0.7500
Ours	0.9625	0.9423

Appendix

Table 24: FireMan-UAV-RGBT fire-presence detection. Scores are on a 0–1 scale. Bold indicates the highest score.

Method	Mean AP	Largest-fire Hit@1	Fire-area NDCG
GPT-5	0.930	0.658	0.947
Claude Sonnet 4.6	0.946	0.850	0.973
Gemini 2.5 Pro	0.952	0.564	0.891
ForestFireVLM-7B	0.939	0.767	0.957
Qwen2-Wildfire-2B	0.844	0.547	0.883
Ours	0.950	0.850	0.984

Appendix

Table 25: FLAME fire-area evaluation. Scores are on a 0–1 scale. Bold indicates the highest score, including ties.

Figure 10: Additional generated wildfire video frames conditioned on 2D simulations through procedural 3D proxies.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Apr 22, 2026

WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring

Wildfire monitoring requires timely, actionable situational awareness from airborne platforms, yet existing aerial visual question answering (VQA) benchmarks do not evaluate wildfire-specific multimodal reasoning grounded in thermal measurements. We introduce WildFireVQA, a large-scale VQA benchmark for aerial wildfire monitoring that integrates RGB imagery with radiometric thermal data. WildFireVQA contains 6,097 RGB-thermal samples, where each sample includes an RGB image, a color-mapped thermal visualization, and a radiometric thermal TIFF, and is paired with 34 questions, yielding a total of 207,298 multiple-choice questions spanning presence and detection, classification, distribution and segmentation, localization and direction, cross-modal reasoning, and flight planning for operational wildfire intelligence. To improve annotation reliability, we combine multimodal large language model (MLLM)-based answer generation with sensor-driven deterministic labeling, manual verification, and intra-frame and inter-frame consistency checks. We further establish a comprehensive evaluation protocol for representative MLLMs under RGB, Thermal, and retrieval-augmented settings using radiometric thermal statistics. Experiments show that across task categories, RGB remains the strongest modality for current models, while retrieved thermal context yields gains for stronger MLLMs, highlighting both the value of temperature-grounded reasoning and the limitations of existing MLLMs in safety-critical wildfire scenarios. The dataset and benchmark code are open-source at https://github.com/mobiiin/WildFire_VQA.

Mobin Habibpour, Niloufar Alipour Talemi, John Spodnik +2

Clemson University

Thermal Imaging Visual Question Answering Remote Sensing VQA Wildfire Detection Multimodal QA Remote Sensing+2

Ask PDF

Aug 24, 2026

Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.

Ryuki Hyodo

SpaceData Inc., 1-17-1 Toranomon, Minato-ku, Tokyo 105-6490, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo 171-8501, Japan · Earth-Life Science Institute, Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan +1

Agent-Based Modeling Large Language Model-Based Social Simulation Vision-Language Agents Multi-Agent LLMs

Ask PDF

Jun 25, 2026

FlameVQA: A Physically-Grounded UAV Wildfire VQA Benchmark with Radiometric Thermal Supervision

Wildfire monitoring from UAVs requires reliable reasoning over complex aerial scenes, where smoke, scale variation, and occlusions often limit RGB-only interpretation. We introduce FlameVQA, a multiple-choice visual question answering benchmark for UAV-based wildfire intelligence built on FLAME 3, leveraging paired RGB imagery and radiometric thermal TIFFs for temperature-grounded, safety-critical reasoning. FlameVQA includes 34 multiple-choice questions per image spanning six operational capability groups, covering tasks such as detection, localization, distribution/coverage estimation, cross-modal reasoning, and flight planning. To ensure label reliability, we combine MLLM-assisted annotation with deterministic thermal rules and cross-question consistency checks, followed by human auditing. We also evaluate representative MLLMs on FlameVQA to provide baselines for future work. Results show strong performance when explicit cross-modal cues are available, but notable failures on presence detection under heavy smoke and on coverage estimation. These findings suggest that current MLLMs require domain-specific adaptation to better support disaster and wildfire monitoring. The dataset and benchmark code are open-source at github.com/mobiiin/WildFire_VQA

Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi +1

Clemson University · Clemson, SC, USA

VLM Evaluation Visual Question Answering Remote Sensing VQA Wildfire Detection Multimodal QA Remote Sensing+2

Ask PDF

Abstract

Vision-Language Models

Explore similar work

WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring

Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

FlameVQA: A Physically-Grounded UAV Wildfire VQA Benchmark with Radiometric Thermal Supervision