AI4Fire: Evaluating Large Language Models on Wildfire Tasks

Yue Zhao; Xiyang Hu; Zuobin Xiong; Zhangyu Wang; Ruolin Li

cs.CLOct 7, 2026

AI4Fire: Evaluating Large Language Models on Wildfire Tasks

Authors: Yue Zhao, Xiyang Hu, Zuobin Xiong, Zhangyu Wang, Ruolin Li

Organizations: University of Southern California · Arizona State University · University of Nevada, Las Vegas · University of Maine

Abstract

Large language models (LLMs) are entering wildfire management, where overstated evaluations can cost property and lives. How do they perform on wildfire tasks, with and without grounding? Bare means a model receives the task input alone. Grounded means it also receives one task-specific addition: for smoke detection, a smoke-free reference frame from the same camera. AI4Fire runs six core models bare and grounded on five wildfire tasks, zero-shot; a sweep adds 29 more. Our literature search on fire tasks found 138 works; none combines this roster, task coverage, and paired bare and grounded runs. We report three findings. (1) Grounding helped most where the addition carried the answer: a read-only SQL tool lifted every core model's database accuracy from at most 16 to at least 88 percent. (2) Simple rules were hard to beat: no core model outperformed repeating today's staffing count, and two open-weight models mostly copied the median of similar earlier fire-days, a worse forecast. (3) Public releases carry hazards: a fire-danger column separates the holdout perfectly, and 67 aerial fire frames carry smoldering or fire-free labels read from a clipped thermal maximum. We release prompts, responses, scores, code, and the survey record.

Figures & tables

Figure 1: Task-specific grounding can supply answers or worsen forecasts; source audits reveal label defects. (a) Each model answers every item bare and grounded ( $g_{t}$ : the task’s addition; $\theta_{m}$ : the task metric); the fire scene is illustrative. (b) Bands span the core six’s lowest to highest accuracy in each arm, not confidence intervals. (c) On Parker Ridge, the allocation example of Appendix K.1 , both grounded open-weight models predict 108, today’s count times the displayed median ratio, rounded. Ratio-rule matches are counted over the 300 rule-v1 items (gpt-6-astra: 199) and do not establish internal reasoning. (d) Actual FLAME 3 inputs and the released CMR4 label for WildFireVQA frame Sycan 2D 00022 ( Habibpour et al., 2026b ) . Its maximum, clipped by an unidentified mechanism, falls below the release’s 200 ∘ C threshold.

	Coverage			Compares
Work	Survey	Cat.	Models	B/G	Ref.
Fire AI reports with no model evaluation
OGC D-123 ( 2024 )
OGC 25-012 ( 2026 )
Fire benchmarks that evaluate no LLM or agent
WILDFIRE-FM ( 2026 )		1			$\bullet$

Table 1: None of the listed prior works combines a re-checked literature survey with a bare-against-grounded evaluation across fire tasks.

\bullet

present,

\circ

survey without claim re-checks, blank absent or unstated. Cat. : fire task categories. Models : configurations evaluated, with families in parentheses as each work reports them (vendors for AI4Fire). B/G : bare and grounded arms on the same items. Ref. : non-LLM comparator. Table A29 adds scale.

Task	Addition	Ans.	Data source	$n$
	Grounded arm
Detection and Perception
Smoke	Reference frame		FIgLib ( 2022 )	196
Aerial QA	Thermal block	$\bullet$	WildFireVQA ( 2026b )	408
Forecasting and Prediction
Fire danger	Climatology		Mesogeos ( 2023 )	386

Table 2: Two of the five additions can carry an item’s answer; the other three give context to reason from. Addition : what the grounded arm adds. Ans. :

\bullet

can carry the reference answer (the SQL tool on every item, the thermal block on 48 closed-form and 13 other items).

n

: scored items (smoke: 196 paired of 224 frames). Appendices D.2 and D.6 give details and licenses.

Metric	Items (clusters)	Comparator	Bare	Grounded	$\Delta$ span	Excl. 0
	Evaluation		Score range, core six		Paired contrast
Personnel allocation, ICS-209-PLUS. Grounding: up to six retrieved analogues (rule v1).
Norm. error $\downarrow$	300 (245 incidents)	Persistence 0.146	0.157–0.193	0.170–0.248	$-0.001$ to $+0.087$	3 $\blacktriangledown$
Smoke detection, FIgLib, first design. Grounding: one earlier clear frame from the same camera.
Smoke recall $\uparrow$	196 (17 fires)	Frame diff. 0.920 †	0.509–0.643	0.554–0.741	$+0.036$ to $+0.188$	4 ∗ $\blacktriangle$
Aerial question answering, WildFireVQA. Grounding: a seven-number thermal summary block.

Table 3: Only the read-only SQL tool clears zero on all six core models. Parentheses give the bootstrap clusters. † A held-out frame-difference detector, shown by recall (clear-frame false-positive rate 0.429, accuracy 0.770; on the 168 rerun targets, 0.726 here and 0.536 once gaps are matched, Appendix I.3 ). Bare , Grounded , and

\Delta

span give the core six’s score ranges and smallest and largest paired differences, as point estimates. Excl. 0 counts models whose 95 percent clustered interval excludes zero (

\blacktriangle

grounded,

\blacktriangledown

bare,

\circ

neither). ∗ Widened 11.5 percent as a finite-cluster check over 17 fires, with no nominal coverage. Other comparators: trained regressor 0.144 (allocation), hybrid rule 0.664 (aerial), boosted classifier 0.851 (fire danger), reference SQL 1.000 (tool use).

Figure 2: The reference frame, retrieved analogues, monthly climatology, and read-only SQL tool move four tasks in different directions. Paired grounded-minus-bare differences with 95 percent intervals clustered as in Table 3 ; coral intervals exclude zero; gray ones include it. (a) Smoke (FIgLib, first design; rerun in Appendix I.3 ): recall gain (circles) and false-positive rate change (mint triangles). (b) Allocation (ICS-209-PLUS): normalized-error difference under retrieval rules v1 (circles) and v2 (squares), which also matches recent staffing change and fire growth (Appendix D.12 ). (c) Fire danger (Mesogeos Track A): average precision difference. (d) Tool use (FPA-FOD): accuracy difference. Figure 3 shows aerial QA.

Figure 3: The thermal summary block leaves pooled accuracy near the majority comparator while splitting closed-form questions from the rest. (a) Accuracy on 408 items with frame-clustered 95 percent intervals; dashed: 0.627 majority. Every paired grounded-minus-bare interval includes zero. (b) 48 closed-form (squares) and 360 others (circles), same scale; bare hollow, grounded filled; shading joins pairs; dotted: 1.000 closed-form rule.

Entry (task)	Evidence
Model failure patterns
Median copying (allocation)	Open-weight forecasts copy the displayed analogue median on 284 and 259 of 300 items
Guessed date format (tool use)	30 of the open-weight models’ 35 failures filter dates in a guessed format
Properties of the benchmark or a source
Unit gloss (aerial, ours)	Our prompt calls the percent fields shares of pixels; 7 of the 9 misses the block added fit a fraction reading
Clipped maximum (aerial)	15 of 34 misses, on frames clipped at 187.4 ∘ C and labeled smoldering or fire-free

Table 4: Selected model failure patterns and properties of the benchmark or a source. Counts cover the core six, and aerial counts are grounded misses on the 48 closed-form items; Appendix I.1 defines each pattern, gives examples, and adds two more.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Grouping	Value	Works
Paper kind
Kind	system-with-eval	75
Kind	evaluation	21
Kind	benchmark	14
Kind	survey	14
Kind	position	8

Appendix

Table A1: Search-unit tags over the 138 kept works: kind, task category, and, for the 90 works a keyword screen marks as LLM work with an evaluation, models tested. Provisional counts that were never re-checked; a work can carry several task tags.

Work	What It Does	What It Leaves Undone
Hyun et al. (2025)	Benchmarks four multi-agent frameworks using GPT-4o in a procedurally generated wildfire environment.	Evaluates one task family (operations in simulation), tests one base model, and provides no literature survey.
Zhang et al. (2026)	Evaluates 21 baseline MLLMs plus DisasterVL on nine UAV disaster-response tasks, including wildfire propagation.	Focuses on general disasters using UAV imagery only; contains no survey of fire literature.
Chen et al. (2025a)	Evaluates about 14 LLM configurations on wildfire personnel and cost allocation, with and without geospatial grounding.	Covers only two related decision tasks.
Tricomi (2024)	Formulates domain adaptation, National Incident Management System and Incident Command System (NIMS/ICS) operational lifecycle mapping, and governance frameworks for generative AI and agent workflows in wildland fire management.	No model evaluation across tasks.
Open Geospatial Consortium (2026)	Assesses over 200 Canadian wildfire datasets for generative AI readiness and outlines prototype architectures for LLMs and agents.	No model evaluation across tasks.
Xu et al. (2026)	Compares a wildfire foundation model with ten Earth foundation models across six task forms (occupancy, spread, burned area, analog retrieval, smoke PM 2.5 , and extreme heat) under a fixed evaluation contract.	No LLMs or agents.

Appendix

Table A2: Our searches found no work that both surveys LLMs and agents across fire tasks and evaluates one model set under one protocol. Closest works found and what each leaves undone relative to this paper.

Task or Source	Category	Domain	Verdict	Access	License
Evacuation guidance dialog, BEACON	Communication and Alerts	wildfire	not feasible now	none found; Watch Duty feed closed to automated access	paper CC BY 4.0; no data license
Crisis translation and urgency classification	Communication and Alerts	both	needs work	public download for three of four corpora	CC BY 4.0 on the one fire-bearing multilingual corpus
Fire database and tool use, FPA-FOD and public APIs	Data Retrieval and Tool Use	both	needs work	public download or free API for all five sources	usable without additional permissions or fees
Fire radiative power forecasting, FIRMS	Forecasting and Prediction	wildfire	needs work	paper table unreleased; FIRMS behind a free key	NASA open data; no license on the paper table
Wildfire risk rasters, FireScope-Bench	Forecasting and Prediction	wildfire	needs work	public download, 127 GB	CC BY 4.0 plus upstream provider terms
Risk sub-criteria scoring against expert ranks	Forecasting and Prediction	wildfire	needs work	public repository download, no registration	article CC BY 4.0; repository states none

Appendix

Table A3: Of 34 checked candidate tasks and sources, three are ready, 20 need work, and 11 are infeasible primarily from lack of released data. The 34 task and source candidates checked on 2026-09-15, with category, fire domain, verdict, access route, and license. Task or Source names the candidate; Category gives the working task category; Domain follows the wildfire and structure split; Verdict marks readiness; Access notes availability; License records redistribution terms.

Task or Source	Category	Domain	Verdict	Access	License
Foundation-model transfer tasks, WILDFIRE-FM	Forecasting and Prediction	wildfire	not feasible now	anonymous repository serves code, metadata, and paper-result files; the Hugging Face link returned HTTP 401 to anonymous requests on 2026-09-18; raw inputs and large feature arrays are excluded	paper CC0; code MIT; source data under provider terms
Fire code compliance, RegCheck audit set	Knowledge Question Answering	structure fire	needs work	audit set none found; code text public	audit set none stated; Approved Document B under OGL v3.0
Fire science and safety question answering	Knowledge Question Answering	both	needs work	public download for all four source corpora	OGL v3.0 and CC BY 4.0
Barriers to fire spread, WFDSS incident text	Document Understanding	wildfire	not feasible now	restricted for sensitivity	article CC BY-NC-ND 4.0; no data license
Incident-command transcript task state	Document Understanding	structure fire	needs work	public repository download, no registration	none stated on the repository
Bushfire damage assessment reports	Document Understanding	wildfire	not feasible now	no dataset released	code CC BY-NC-ND 4.0, which blocks derivatives

Appendix

Table A4: (continued)

Task or Source	Category	Domain	Verdict	Access	License
Operations by incident-command phase, OGC 24-071	Decision Support and Operations	wildfire	not feasible now	report public; no evaluation items in it	OGC document license; NWCG source text public domain
Risk and insurance retrieval, OGC 25-012	Decision Support and Operations	wildfire	not feasible now	report public; no evaluation items in it	OGC document license; no license given for listed sources
Suppression and rescue, CREW-Wildfire	Simulation-Coupled Agents	wildfire	ready	public download; environment and scoring released	code Apache-2.0; paper CC BY-NC-ND 4.0
Policy adaptation, MOASEI and free-range-zoo	Simulation-Coupled Agents	wildfire	needs work	public download; frozen evaluation tag	MIT on the environment; AGPL-3.0 on the starter
Zero-shot active fire classification, Roboflow set	Detection and Perception	wildfire	not feasible now	Roboflow download with a free account	CC BY 4.0
Smoke classification and localization, SmokeBench	Detection and Perception	wildfire	needs work	none found for the benchmark; FIgLib images public	preprint CC BY 4.0; benchmark none

Appendix

Table A5: (continued)

Task or Source	Category	Domain	Verdict	Access	License
Satellite risk reasoning, Wildfire-Smoke-RS	Detection and Perception	wildfire	not feasible now	none found; the repository holds only a README	none stated
UAV disaster reasoning, DisasterBench	Detection and Perception	both	needs work	public download of the test split	Apache-2.0
Smoke sequences, FIgLib	Detection and Perception	wildfire	ready	public download, no registration	FIgLib page names no license and requires credit; HPWREN data-use page lists CC BY-NC-ND 4.0
Fire-scene severity items, DetectiumFire	Detection and Perception	both	not feasible now	Kaggle download with a free account	two authoritative sources disagree, both non-commercial
Structure-fire video questions, Fire360	Detection and Perception	structure fire	needs work	public Box download, no login	paper: MIT with research-only restrictions; released license file: standard MIT text
Post-fire change captions, RSCC	Detection and Perception	wildfire	not feasible now	public download, not gated	paper: CC BY 4.0; repository binds the xBD-derived subset to the xBD terms and research use

Appendix

Table A6: (continued)

Source	What reproduces	What it does not establish
Mesogeos Track A	A released static column separates the holdout at average precision 1.000; a rule over two window columns reaches 0.814 on the last day	That the published baselines read those columns. All three released configurations name their inputs and exclude them
Fire360	The released 100 questions carry no reference to a video, a frame, or a timestamp, and the answer key puts the correct choice first on 25 of 50	That the released items are the scored ones. The release calls them sample prompts, and no mapping ties them to the reported evaluation
WildFireVQA	Four question types are reproduced exactly by closed-form rules on the supplied thermal block, on 6,097 of 6,097 frames each. On 67 frames filed as fire, the maximum is clipped at 187.4 ∘ C, and every temperature-derived label reads smoldering or no fire	That the block is illicit, that a model missing these labels ignored the block, or where the clipping arises. The source sets out to study thermal retrieval; what is missing is a text-only arm beside the pooled score
This benchmark	A few-hundred-token output cap truncated a reasoning model’s answer on 40 of 196 items, against 8 for another family	That the affected runs measure model skill. The cap selected which items each family was scored on

Appendix

Table A7: Source hazards found while building the benchmark, in three other releases and in ours.

		Allocation nMAE		Fire danger AUPRC		Fire-call rate		Tool use acc.
Model	Cap	bare	grd.	bare	grd.	bare	grd.	bare	tool
Nova Micro	1,536	0.166	0.289	0.339	0.332	0.000	0.000	0.019	0.821
Llama 3.3 70B	1,536	0.927	0.248	0.413	0.458	0.446	0.440	0.090	0.288
Llama 3.1 70B	1,536	0.205	0.257	0.400	0.427	0.145	0.202	0.006	0.000
Mistral Small 2402	1,536	0.491	1.958	0.401	0.407	0.858	0.671	0.058	0.038
Devstral 2 123B	1,536	0.179	0.237	0.522	0.522	0.681	0.674	0.096	0.872

Appendix

Table A8: The nineteen text-only models on the three text tasks. Point estimates from single runs at temperature zero on the same items and scorers as the main tables; the cap column is the output cap in tokens. Allocation: nMAE (persistence 0.146). Fire danger: AUPRC (temperature rule 0.654), call rate (sample positive rate 0.339). Tool use: accuracy bare and with the tool (best constant per family 0.141); a dash marks the two models whose tool probe failed. MiniMax M2.5 covers 278 of 300 bare and 296 of 300 grounded allocation items, and 312 of 386 bare and 300 of 386 grounded fire-danger items. Persistence on those allocation subsets is 0.143 and 0.146; tool accuracy counts all 156 items. Mistral Small 2402’s grounded error is 1.958, and 0.520 without its 866-person prediction for a two-person incident. Gemma 3 4B, in Table A28 , returns the previous day’s count on 299 of 300 bare items.

Figure A1: Across the nineteen text-only models, intervals exclude zero for the analogues’ error increase on seventeen and the SQL tool’s lift on fifteen of seventeen; climatology’s AUPRC change excludes zero on three, in both directions. Paired grounded-minus-bare differences (tool minus bare in c) with 95 percent cluster-bootstrap intervals, clustered as in Table 3 , on the items each model parsed in both arms (in c, all 156 items, with an unanswered response counted wrong); coral intervals exclude zero and gray ones include it. (a) Allocation: normalized-error difference. Llama 3.3 70B (

-0.68

) and Mistral Small 2402 (

+1.47

) lie beyond the axis, each moved by one extreme prediction (Table A8 ; Appendix I.7 ), and both intervals contain zero. (b) Fire danger: average precision difference; MiniMax M2.5’s interval lies below zero. (c) Tool use: accuracy difference; Llama 3.1 8B and DeepSeek R1 failed the tool probe. Figure A7 shows the sixteen full-capability models.

Model	Condition	Mean $p$	ECE	Brier	Reliability	Resolution	Departures	Omitted
Reference points
Calendar-month prior	–	0.333	0.037	0.182	0.003	0.046	–	–
Proprietary models
claude-opus-4.8	bare	0.187	0.154	0.206	0.034	0.051	50	0
claude-opus-4.8	grounded	0.220	0.119	0.200	0.016	0.040	50	2
claude-opus-5	bare	0.117	0.222	0.230	0.069	0.061	11	46

Appendix

Table A9: Every proprietary model underpredicts fire frequency on the benchmark sample, and the calendar-month prior has a lower ECE than every model run. Calibration of the stated probabilities on fire danger forecasting, 386 items, sample positive rate 0.339. Condition is bare or grounded; Mean

p

is the mean stated probability; ECE is the expected calibration error over ten equal-width bins. Reliability and Resolution are the two Murphy terms computed after each probability is replaced by its within-bin mean, decomposing the binned Brier score, which differs from the raw Brier column by up to 0.005; lower reliability and higher resolution are better. Departures counts items whose yes or no answer differs from the stated probability thresholded at one half; Omitted counts items with a probability and no answer. The calendar-month prior is fitted on the 2006 to 2019 training years.

Model	Condition	Effective	ECE	Brier	Rel.	Res.	AUPRC	Brier minus prior
		prior						and 95% CI
As stated
Calendar-month prior	–	0.331	0.037	0.182	0.003	0.046	0.555	–
claude-opus-4.8	bare	0.149	0.154	0.206	0.034	0.051	0.613	0.024 [ $-0.001$ , 0.049]
claude-opus-4.8	grounded	0.184	0.119	0.200	0.016	0.040	0.581	0.018 [ $-0.008$ , 0.043]
claude-opus-5	bare	0.083	0.222	0.230	0.069	0.061	0.688	0.048 [0.016, 0.079]

Appendix

Table A10: After a prior-shift correction, no proprietary run has a resolvably worse Brier score than the calendar-month prior, and both open-weight models stay worse. The fire-danger probabilities of the twelve core-model runs on the 386 items of Table A9 (rate 0.339, 352 blocks), as stated and rescored two ways, with the calendar-month prior treated alike. As stated reproduces Table A9 ; Effective prior is the prior that the fitted odds constant implies if a run’s probabilities were calibrated under a pure label shift; it need not describe what the model believes. Prior-shifted multiplies each run’s odds by the one constant that makes its probabilities average 0.339 ( Saerens et al., 2002 ) . Recalibrated applies isotonic regression fit on four folds to the fifth, with blocks kept whole (five folds, seed 7). ECE uses ten equal-width bins; Rel. and Res. are the binned reliability and resolution terms of Table A9 , and the uncertainty term is 0.224 in every row. Brier minus prior compares each run with the prior under the same treatment by a paired 95 percent block bootstrap (20,000 resamples, seed 20260915). Recalibrated rows refit the recalibration in every resample with folds fixed at seed 7, and only gemini-3.1-pro bare resolves in all 20 fold draws; bold intervals exclude zero. The shift preserves order, so AUPRC is unchanged; isotonic maps tie neighboring values and differ by fold, which lowers AUPRC.

Model	Arm	Mean $p$	First	Rerun	Brier	Brier minus prior	Calls	AUPRC
			ECE
Calendar prior	–	0.333	0.037	–	0.182	–	–	0.555
claude-opus-4.8	bare	0.187 $\to$ 0.305	0.154	0.068	0.206 $\to$ 0.179	$-0.004$ [ $-0.021$ , 0.014]	0.215 $\to$ 0.311	0.613 $\to$ 0.607
claude-opus-4.8	grounded	0.220 $\to$ 0.355	0.119	0.076	0.200 $\to$ 0.189	+0.006 [ $-0.011$ , 0.024]	0.249 $\to$ 0.399	0.581 $\to$ 0.581
claude-opus-5	bare	0.117 $\to$ 0.267	0.222	0.087	0.230 $\to$ 0.168	$-0.015$ [ $-0.035$ , 0.006]	0.065 $\to$ 0.174	0.688 $\to$ 0.701
claude-opus-5	grounded	0.116 $\to$ 0.289	0.224	0.066	0.232 $\to$ 0.173	$-0.010$ [ $-0.030$ , 0.010]	0.109 $\to$ 0.241	0.682 $\to$ 0.663

Appendix

Table A11: Stated in the prompt, the 0.339 rate calibrates the proprietary four about as well as the prior shift, and leaves both open-weight models worse than the calendar-month prior. Fire danger on the 386 items of Table A10 . Each core run is shown as first run to rerun with one added line stating the sample rate. Brier minus prior : rerun Brier score minus that of the calendar-month prior shifted to 0.339 (0.183), with a 95 percent block-bootstrap interval (352 blocks, 20,000 resamples); * marks an interval excluding zero. Calls : share of items called fire as scored, where an omitted call counts as

p\geq 0.5

. One run per model and arm.

Fit	MAE	Norm.	Beats	Stable	Moving	False	Missed	Dir.
			base	days	days	move	move	right
Reference points
Boosted regressor, next-day ratio	12.97	0.144	0.48	0.035	0.317	0.03	0.93	0.72
Same, plus the displayed analogue median	13.03	0.145	0.48	0.035	0.319	0.03	0.94	0.71
Boosted regressor, next-day count	13.68	0.201	0.42	0.126	0.319	0.14	0.81	0.66
Ridge regression, next-day count	19.05	0.416	0.28	0.412	0.423	0.63	0.47	0.57

Appendix

Table A12: A boosted regressor on the bare prompt numbers reaches 0.144 normalized error on allocation, matching persistence without gaining from the displayed analogues. Trained regressors on daily personnel allocation, in the columns of Table A23 , on the same 300 items. Fit specifies the model and target; MAE is mean absolute error; Norm. is MAE normalized by incident mean staffing; Beats base is the share of items beating persistence; Stable days and Moving days give normalized error by true movement; False move and Missed move are directional error shares; Dir. right is the share moving the correct direction. Each fit uses 30,869 fire-days from 1,499 incidents disjoint from the evaluation incidents and reads the 23 bare-prompt fields; the second row also reads the displayed analogue median.

Model	Prompt	AUPRC	F1 on fire	Call rate	Omitted	Difference in AUPRC	Spearman
Proprietary models
claude-opus-4.8	p0	0.613	0.561	0.215	0
claude-opus-4.8	p1	0.605	0.234	0.060	0	$-0.008$ [ $-0.053$ , 0.035]	0.93
claude-opus-4.8	p2	0.631	0.059	0.010	0	0.018 [ $-0.025$ , 0.057]	0.91
claude-opus-5	p0	0.688	0.269	0.065	46
claude-opus-5	p1	0.731	0.308	0.065	0	0.043 [0.009, 0.080]	0.92

Appendix

Table A13: Single runs of two paraphrases reorder the six models: gpt-6-astra rises from third to first, the two leaders swap, and Llama 4 Maverick loses 0.13. Two models nearly stop calling fire. Prompt sensitivity on fire danger forecasting, bare condition, 386 items, one run per paraphrase. Model is the evaluated system. Prompt specifies the paraphrase: p0 is the paper prompt, and p1 and p2 differ in framing, field order, and layout. Difference in AUPRC is AUPRC under the paraphrase minus p0, with a paired block-by-month bootstrap interval. Spearman is the rank correlation of stated probabilities across the two prompts.

Model	Bare	v1	v2	v2 $-$ bare	v2 $-$ v1	False move	Copies
Proprietary models
claude-opus-4.8	0.157	0.170	0.166	0.010	$-0.004$	0.14 / 0.29 / 0.22	123 / 145
				[ $-0.002$ , 0.023]	[ $-0.020$ , 0.014]
claude-opus-5	0.165	0.173	0.166	0.000	$-0.007$	0.25 / 0.27 / 0.20	82 / 93
				[ $-0.011$ , 0.011]	[ $-0.022$ , 0.006]
gemini-3.1-pro	0.179	0.177	0.166	$-0.012$	$-0.011$	0.23 / 0.29 / 0.20	154 / 152

Appendix

Table A14: Under the nearest-neighbour retrieval rule (v2), grounded allocation error falls on every model and false movement decreases across all six. Daily personnel allocation, 300 items, under the paper’s analogue rule (v1) and the frozen nearest-neighbour rule (v2). Bare , v1 , and v2 report normalized error as in Table A23 ; the two difference columns carry incident-clustered 95 percent intervals; False move is the share of stable days moved by more than a tenth, bare / v1 / v2; Copies counts items whose prediction equals persistence times the displayed median ratio, v1 / v2. The analogue-only rules themselves score 0.249 (v1) and 0.190 (v2) against persistence at 0.146.

Proprietary models
Model	Condition	0 to 10 min	10 to 25 min	25 min or more
claude-opus-4.8	bare	0.43	0.57	0.62
claude-opus-4.8	grounded	0.46	0.66	0.68
claude-opus-5	bare	0.50	0.66	0.62
claude-opus-5	grounded	0.50	0.72	0.65
gemini-3.1-pro	bare	0.25	0.64	0.65

Appendix

Table A15: Detection accuracy is lowest in the first ten minutes after the plume across every evaluated model and condition. Accuracy on FIgLib frames after the first visible plume, by offset bucket, on 196 paired items (buckets hold 28, 47, and 37 frames; pre-plume clear frames score 0.94–1.00).

Proprietary models
Model	Condition	CL	CMR	DS	FP	LD	PD
Per-question majority		0.708	0.438	0.573	0.604	0.542	0.771
claude-opus-4.8	bare	0.625	0.562	0.625	0.625	0.521	0.781
claude-opus-4.8	grounded	0.569	0.708	0.604	0.604	0.542	0.771
claude-opus-5	bare	0.694	0.625	0.594	0.646	0.562	0.729
claude-opus-5	grounded	0.681	0.750	0.625	0.646	0.542	0.688

Appendix

Table A16: The thermal block lifts Cross-Modal Reasoning and Distribution on most models while lowering Classification and Presence. Accuracy by category on WildFireVQA across all six models, bare and grounded, beside the majority baseline. Categories: Classification (CL, 72 items), Cross-Modal Reasoning (CMR, 48), Distribution and Segmentation (DS, 96), Flight Planning (FP, 48), Localization and Direction (LD, 48), and Presence and Detection (PD, 96).

Grounded $-$ bare
Model	Diff.	By frame	By question type	Two-way
claude-opus-4.8	$+0.000$	[ $-0.027$ , $+0.027$ ]	[ $-0.039$ , $+0.056$ ]	[ $-0.056$ , $+0.071$ ]
claude-opus-5	$+0.007$	[ $-0.020$ , $+0.035$ ]	[ $-0.037$ , $+0.064$ ]	[ $-0.050$ , $+0.079$ ]
gemini-3.1-pro	$+0.015$	[ $-0.015$ , $+0.044$ ]	[ $-0.025$ , $+0.059$ ]	[ $-0.040$ , $+0.077$ ]
gpt-6-astra	$+0.012$	[ $-0.012$ , $+0.037$ ]	[ $-0.012$ , $+0.039$ ]	[ $-0.030$ , $+0.056$ ]
Qwen3-VL	$+0.000$	[ $-0.030$ , $+0.030$ ]	[ $-0.039$ , $+0.042$ ]	[ $-0.057$ , $+0.060$ ]

Appendix

Table A17: The pooled arm contrast holds under every resampling; one comparator result does not. Paired contrasts on the 408 aerial items under three bootstrap designs: clustering on the 390 frames, clustering on the 34 question types, and a two-way pigeonhole bootstrap over both. All use 20,000 resamples at seed 20260915. Grounded

-

bare is the pooled arm contrast. Bare

-

majority is the contrast against the held-out per-question majority comparator of 0.627.

Work	Category	Domain	Models confirmed	Bare or grounded	Headline number
RegCheck-Hybrid ( Chen and Zhou, 2025 )	Knowledge QA	Structure	Rule-LLM hybrid, RAG LLM baseline	Both	89% F1 on building fire code compliance
Triple-Phase control ( Sun et al., 2026 )	Knowledge QA	Structure	Commercial RAG LLMs, pure LLMs	Both	Hallucination on numerical thresholds eliminated
WFDSS barrier analysis ( Epstein and Seielstad, 2025 )	Documents	Wildfire	GPT-4 (1)	Bare	13 barrier types over 24,254 entries; roads 42%
Task-state monitoring ( Grünert et al., 2026 )	Documents	Fire service	gpt-5.2, gpt-5.4 (2)	Bare	Strong incrementally; unit-assignment timing weakest
DisasTeller ( Chen et al., 2026 )	Documents	Multi-hazard	GPT-4o, Gemma3 (2)	Grounded	Damage reports on earthquake, flood, and bushfire
Fire protection specifications ( Kim and Shin, 2025 )	Documents	Structure	Gemini, Claude, ChatGPT (3)	Bare	Adequate structures; expert review still required

Appendix

Table A18: Confirmed findings, confirmed model counts, and grounding status across the 25 re-checked fire records. One row per record, grouped by the task category its confirmed finding measures. Multi-hazard marks a disaster-wide evaluation that includes a fire task; Both marks a study that compares a bare condition against a grounded one.

Work	Category	Domain	Models confirmed	Bare or grounded	Headline number
DisasterBench ( Zhang et al., 2026 )	Detection	Multi-hazard	21 baseline MLLMs plus DisasterVL	Bare	2B DisasterVL matched GPT-4o at 72.60%
WildfireGPT assessment ( Ramesh et al., 2025 )	Forecasting	Wildfire	WildfireGPT (GPT-4), TabNet (2)	Grounded	MAE 14.849 and $R^{2}$ 0 against MAE 0.055
FireScope ( Markov et al., 2026 )	Forecasting	Wildfire	Qwen2.5-VL-7B, GPT-5 (3)	Grounded	Out-of-distribution transfer from US to Europe
Retrieval over Response ( Cheng et al., 2026 )	Forecasting	Wildfire	Not listed	Both	$R=0.009$ direct against $R=0.598$ guided
IC-EO ( Lahouel et al., 2026 )	Geospatial	Wildfire	IC-EO agent, GPT-4o, LLaVA (3)	Both	50% against 0% for GPT-4o and LLaVA
Geospatial-aware agents ( Chen et al., 2025a )	Operations	Wildfire	About 14 LLM configurations	Both	Text-only over-allocated personnel 2–3 times

Appendix

Table A19: (continued)

Category	Definition	Example Re-Checked Fire Works
knowledge-qa	Fire science, engineering, codes, or safety question answering.	( Chen and Zhou, 2025 ; Sun et al., 2026 )
document-understanding	Incident reports, after-action reviews, records, and extraction.	( Epstein and Seielstad, 2025 ; Grünert et al., 2026 ; Kim and Shin, 2025 ; Tao et al., 2026 ; Chen et al., 2026 )
detection-perception	Fire or smoke recognition from imagery or video by VLMs or MLLMs.	( Seidel et al., 2025 ; Qi et al., 2026 ; Kohli, 2025 ; Habibpour et al., 2026b ; Ayanzadeh et al., 2026 ; Zhang et al., 2026 ; Al-Mohannadi et al., 2026 )
forecasting-prediction	Spread, risk, danger, occurrence, or behavior prediction by LLMs or agents.	( Ramesh et al., 2025 ; Cheng et al., 2026 ; Markov et al., 2026 )
geospatial-analysis	GIS reasoning, spatial computation, map or remote-sensing interpretation.	( Lahouel et al., 2026 )
decision-support-operations	Resource allocation, dispatch, evacuation, and incident command.	( Chen et al., 2025a ; Open Geospatial Consortium, 2026 )

Appendix

Table A20: Ten working task categories organize the survey from perception to operations. Assignments are provisional; Appendix A.4 records boundary decisions.

Figure A2: The 138 kept works by kind and task category, and the number of models the evaluated ones test. (a) Task category by kind of work over the 138 kept works (316 tags, since a work can carry several); coral marks the largest cell, the 39 systems with an evaluation tagged decision-support-operations. (b) Number of models tested by the 90 works that a keyword screen marks as LLM work with an evaluation. Every count is a search-unit tag that was never re-checked.

Category	$n$	What the records show	What they do not settle
Detection and perception	7	Binary fire recognition scores high; early-stage smoke and quantitative estimates do not	The datasets fix camera set, prevalence, and smoke stage; no study shares a model set with another
Document understanding	5	Models sort incident records, command transcripts, and investigation cases at working scale	None of the five reports an error rate validated against human agreement
Operations and allocation	3	Geospatial grounding improved personnel and cost allocation	One study against baselines its authors chose; filed staffing is a proxy for decision quality
Forecasting and prediction	3	The one direct comparison put an LLM agent below tabular deep learning on fire radiative power	A single comparison on a GPT-4-era model; no ranking against other categories follows
Simulation-coupled agents	2	Coordination held on small suppression tasks and broke down on large ones	One base model; the second record is a separate result rather than a replication
Knowledge question answering	2	Rule-first and retrieval systems removed hallucination on numerical thresholds	Both are structure-fire code tasks, each against baselines its authors chose

Appendix

Table A21: What the 25 re-checked fire records show, by task category: what they report, and what they leave unsettled. Every row describes the re-checked subset alone, so no column is a prevalence estimate; Table A18 in Appendix E gives one row per record.

Survey pattern (Section 2 )	Design decision	Result (Section 4 )
Grounding helped in each re-checked study that tried it	Same models bare and grounded on every task, with an information-only comparator per source	Where an addition carried the answer and reached the model, every resolved difference was a gain; additions lacking it moved scores in both directions, and surveyed systems are not replicated (Section 4 )
Recognition high; early smoke and quantities weak	Smoke frames scored by minutes since the plume	Smoke accuracy is 0.25 to 0.57 in the first ten minutes (Section 4.1 ). On aerial question answering the thermal block moved pooled accuracy by at most 0.015, and no core model’s paired interval against the 0.627 majority comparator lies wholly above zero (Section 4.1 )
One direct forecasting comparison favored a tabular model	Trivial rules and published reference points beside every model	Four of twelve prompted configurations beat a temperature rule, and trained tabular classifiers beat all twelve (Section 4.1 )
Evaluations hard to reconstruct; two records test models behind the frontier	Access, reference answers, and leakage audited; every response stored; frontier models of the run date	Three hazards in public releases and one in our harness (Section 3.5 ); 11 of 34 candidate sources not feasible

Appendix

Table A22: Four survey patterns, the design decision each led to, and what the tasks returned.

Reference points
Model	Condition	MAE	Norm.	Beats	Stable	Moving	False	Missed	Dir.
				base	days	days	move	move	right
Persistence baseline		13.76	0.146		0.027	0.337	0.00	1.00
Analogue-only rule		19.88	0.249	0.23
Trained ratio regressor		12.97	0.144	0.48	0.035	0.317	0.03	0.93	0.72
Proprietary models

Appendix

Table A23: No prompted model beats persistence in normalized error; the v1 analogues raise it on five of six models. Daily personnel allocation on ICS-209-PLUS, 300 items, all answered. Persistence repeats today’s filed count; analogue-only multiplies it by the median next-day ratio of the analogues the grounded arm saw; the trained regressor fits that ratio on 1,499 other incidents (Appendix D.10 ). The right columns split the items by whether the filed count moved by more than a tenth (116 moving days): false move is the share of stable days moved by more than a tenth, missed move the share of moving days held flat, dir. right the share of moving days moved the same way. Norm. is error normalized by each fire’s mean staffing.

Figure A3: The two open-weight models match the displayed-median rule on most items; rule v1 has a wider interquartile range than the filed next-day ratios, and rule v2 narrows it. (a) Personnel allocation under retrieval rule v1, 300 items per model: the model’s next-day ratio (prediction over persistence) against the displayed median ratio of its six analogues, on

\log_{2}

axes with the diagonal

y=x

dashed. Coral marks a prediction equal to persistence times the displayed median, rounded to an integer, and each panel prints the count of such items. (b) Across the 300 items, the filed next-day ratio (gray), the displayed median under rule v1 (coral), and the displayed median under rule v2 (mint): boxes span the quartiles, the bar is the median, whiskers run from the 5th to the 95th percentile, and the dashed line is persistence.

Reference points
Model	Condition	Items	Accuracy	Recall on smoke	FPR	Sequences detected
Constant: always smoke		196	0.571	1.000	1.000	28 of 28
Constant: always clear		196	0.429	0.000	0.000	0 of 28
Frame difference, held out		196	0.770	0.920	0.429	28 of 28
Proprietary models
claude-opus-4.8	bare	196	0.745	0.554	0.000	23 of 28

Appendix

Table A24: The reference frame raises smoke recall on every model and false positives on four. Wildfire smoke detection on FIgLib, scored per model on the 196 items both of its conditions answered; no item was lost to truncation. Sequences detected counts the smoke sequences in which at least one frame was called. The frame-difference detector thresholds the mean absolute difference against the reference frame, with the cut chosen by leave-one-fire-out cross-validation.

Figure A4: Detection is lowest in the first ten minutes after the plume. The reference frame adds about eight to nine accuracy points in each post-plume window, pooled over models. (a) Accuracy per offset bucket for each of the six models, bare (gray) and grounded (coral), on the 196 frames per model that both conditions answered; the frame counts under the axis are per model, and the three buckets before the plume are pooled into one. (b) The grounded-minus-bare accuracy difference per bucket, one coral point per model and a black bar pooled over the six, with the pooled difference and the correct decisions it adds, summed over the six models, printed above each bucket. No intervals are drawn; Section 4.1 reports the fire-clustered intervals for the whole-task differences.

Model	Condition	AUPRC	F1 on fire	Call rate	Omitted
Reference points
Calendar-month prior, training years		0.555
Last-day maximum 2 m temperature		0.654
Published baselines, full holdout		0.853 to 0.858
Trained boosted classifier, prompt numbers		0.851	0.783	0.316	0
Trained logistic regression, prompt numbers		0.768	0.697	0.285	0

Appendix

Table A25: Two models rank above the temperature rule, and a classifier trained on the prompt’s own numbers ranks above all twelve. Fire danger forecasting on Mesogeos Track A, 386 items, sample positive rate 0.339. The temperature rule is the best trivial strategy found; the trained rows read the 121 numbers the bare prompt prints (Appendix D.10 ); call rate is the share of items answered yes; omitted counts items with a probability but no yes or no answer, filled from the probability at one half. In claude-opus-5’s grounded arm those omissions fall on 29 percent of non-fire items against 12 percent of fire items (

p=0.0002

, two-sided Fisher exact test), so its grounded decision column rests on that imputation. Its bare rates are 13 and 11 percent, with no resolved association (Appendix D.7 ).

Figure A5: Every proprietary model states mean probabilities of 0.06 to 0.22 against the 0.339 benchmark sample positive rate, underpredicting fire on this selected population. The calendar-month prior has a lower ECE than all twelve model runs. Reliability diagrams on the 386 fire-danger items: the mean stated probability in each of ten equal-width bins against the observed fire frequency in that bin, bare in gray and grounded in coral; marker area is

\max(0.35n,3)

square points for a bin of

n

items, so bins of fewer than nine items share the smallest marker, and empty bins are omitted; the dashed diagonal is perfect calibration. The calendar-month prior (mint, the same curve on every panel) was fitted on identically sampled training data. Its expected calibration error (ECE) is 0.037 against 0.119 to 0.326 for the twelve model runs. After one odds shift to the sample rate, claude-opus-5 falls below the prior on Brier score in both arms and no other proprietary interval excludes zero (Table A10 ).

Figure A6: Single runs of two paraphrases move the two leading models in opposite directions and reverse their order; the paraphrase reorders gemini-3.1-pro’s items far more than claude-opus-5’s. Stated fire probability on the 386 Mesogeos Track A items under the paper’s prompt (p0, horizontal) against a paraphrase (vertical) for claude-opus-5 (a, b) and gemini-3.1-pro (c, d). For each label, one bubble represents the items that share a distinct pair of stated values, with area proportional to that count; fire items (

n=131

) are coral and no-fire items (

n=255

) gray, and bubbles of the two labels can share a center, with coral drawn above gray. The dashed diagonal marks unchanged probabilities. The subtitle of each panel gives the Spearman correlation between the two runs and the AUPRC under p0 and under the paraphrase. Paired AUPRC differences carry 95 percent intervals from a cluster bootstrap over 1 ∘ blocks by calendar month (20,000 resamples). They are 0.043 [0.009, 0.080] and 0.015 [

-

0.018, 0.049] for claude-opus-5, and

-

0.049 [

-

0.093,

-

0.009] and

-

0.026 [

-

0.075, 0.020] for gemini-3.1-pro (Table A13 , which adds the other four models). Each paraphrase ran once, bare.

Model	Arm	Accuracy	Single lookup	Filtered aggregate	Multi-step	Abstained	Calls	Tool $-$ bare
Reference point
Best constant per family		0.141	0.077	0.077	0.231
Proprietary models
claude-opus-4.8	bare	0.103	0.000	0.000	0.246	99	0.00
claude-opus-4.8	tool	1.000	1.000	1.000	1.000	0	1.13	0.897 [0.756, 1.000]
claude-opus-5	bare	0.160	0.000	0.038	0.338	35	0.00

Appendix

Table A26: One read-only SQL tool lifts every core model from at most 0.160 accuracy to 0.885 or above, and the four proprietary models to 0.994 or above. Most open-weight errors fall in the calendar-window count family. Fire data tool use on FPA-FOD, 156 items in twelve families, one run per arm. Best constant is the one fixed answer per family that scores highest under the family’s tolerance, chosen with the answers in hand. Abstained counts unknown answers, scored as wrong. Calls is the mean number of tool calls per item. Tool

-

bare is the paired accuracy difference, with a 95 percent bootstrap interval clustered by family (12 clusters, 20,000 resamples).

Proprietary models
Model	Original	Phrasing	De-schem.	Phrasing $-$ orig.	De-schem. $-$ orig.
claude-opus-4.8	1.000	1.000	1.000	+0.000	+0.000
claude-opus-5	1.000	1.000	1.000	+0.000	+0.000
gemini-3.1-pro	1.000	1.000	1.000	+0.000	+0.000
gpt-6-astra	0.994	0.994	0.994	+0.000	+0.000
Open-weight models

Appendix

Table A27: Surface rewording moves neither open-weight model’s interval clear of zero; removing schema literals leaves four proprietary models unchanged and costs two open-weight models 0.175 and 0.126 on the clean 143-item subset. Accuracy under original items, a phrasing rewrite, and a de-schematized rewrite. The table reports all 156 items (where open models drop 0.199 and 0.154); the footnote reports the clean 143 items. Intervals expand the clustered 95 percent percentile intervals by the sensitivity procedure of Appendix D.19 . The 39-column schema reaches models under all three.

Core set (Section 3.3 )
	Allocation nMAE		Smoke recall		Fire danger AUPRC		Tool use acc.		Aerial acc.
Model	bare	grd.	bare	grd.	bare	grd.	bare	tool	bare	grd.
claude-opus-4.8	0.157	0.170	0.554	0.616	0.613	0.581	0.103	1.000	0.642	0.642
claude-opus-5	0.165	0.173	0.607	0.643	0.688	0.682	0.160	1.000	0.650	0.657
gemini-3.1-pro	0.179	0.177	0.545	0.732	0.697	0.682	0.160	1.000	0.632	0.647
gpt-6-astra	0.161	0.220	0.643	0.741	0.620	0.590	0.000	0.994	0.623	0.635

Appendix

Table A28: Among the sixteen models evaluated on all five tasks, no bare model beats persistence. Analogues raise allocation error on thirteen, a reference frame raises smoke recall on fifteen, and the tool lifts the fourteen whose interface worked. No added model reaches the aerial majority baseline. The model sweep of 2026-09-18 beside the core six: point estimates per model and arm on the same items and scorers as Tables A23 , A24 , A25 , A26 , and A32 . Allocation: normalized error, lower is better. Smoke: recall on the smoke frames of the items both arms answered, 112 of 196 for fifteen models and 111 of 194 for Mistral Large 3. Fire danger: AUPRC. Tool use: accuracy bare and with the tool. Aerial: accuracy over 408 items. The comparator row holds persistence, the temperature rule, the best constant per family, and the majority baseline. Nova 2 Lite’s grounded allocation error is 0.248 without one item on which it copied an analogue’s 866 personnel onto a two-person incident. Gemma 3 12B and 4B score 0.000 with the tool because neither executed a valid database query under the declared interface (Appendix D.5 ). Appendix D.5 adds the false-positive and fire-call rates.

Figure A7: Across sixteen models, intervals exclude zero for the reference frame’s recall gain on twelve, the analogues’ error increase on nine, and the tool’s lift on fourteen. Climatology’s AUPRC change excludes zero on five models, in both directions. Grounded minus bare per model on four tasks, with 95 percent cluster bootstrap intervals as in Figure 2 ; the ten added models carry retrieval rule v1 only. Individual 95 percent intervals carry no multiplicity adjustment across the sixty-four contrasts, so these counts are descriptive summaries rather than formal joint inferences. Coral marks an interval excluding zero. Nova 2 Lite’s allocation point (+1.52) lies beyond the axis (one analogue copy; Table A28 ), and its recall change is the one negative interval.

Work	Surv.	Tasks	LLMs	Fam.	B/G	Non-LLM	Scale
Fire AI reports with no model evaluation
OGC D-123 ( Tricomi, 2024 )	–	–	–	–	–	–	NIMS/ICS phases
OGC 25-012 ( Open Geospatial Consortium, 2026 )	–	–	–	–	–	–	$>$ 200 datasets
Fire benchmarks and datasets that evaluate no LLM or agent
WILDFIRE-FM ( Xu et al., 2026 )	–	1	–	–	–	✓	6 task forms, 11 FMs
Land8Fire ( Tran et al., 2025 )	✓ $\circ$	1	–	–	–	✓	4 nets, 55 GB

Appendix

Table A29: None of the listed prior works combines a re-checked literature survey with a bare-against-grounded evaluation of several model families across fire tasks. Prior fire surveys, benchmarks, and LLM evaluations, one row each.

	Clear frames			Smoke frames
Gap (min)	10	20	30	45	55	65	79
(a) Changed verdicts by ladder position
claude-opus-4.8	0/0	0/0	0/0	1/0	2/0	3/0	1/0
claude-opus-5	1/0	1/0	1/1	0/0	1/0	2/0	1/0
gemini-3.1-pro	1/0	0/0	1/0	7/0	4/0	4/0	6/0
gpt-6-astra	1/0	0/1	2/0	3/0	3/0	3/0	2/0

Appendix

Table A30: The reference frame gains smoke frames at every gap and its clear-frame false positives stay flat, but the gap alone separates the labels by design. Smoke detection on FIgLib, six core models, one run per model and arm, on the 196 paired items. The reference is each sequence’s frame about 40 minutes before the first visible plume. Gap is the median minutes from the reference to the target at each ladder position, which holds 28 frames per model. (a) Per model, clear columns count new/removed false positives and smoke columns count gained/lost smoke frames, grounded against bare. Pooled rows give the six-model rate or change per position with its 95 percent interval. The mint rows give the held-out frame-difference detector’s false-positive rate (clear) and recall (smoke). (b) Least-squares slope, per 10 minutes of gap, of the paired change (grounded minus bare) over smoke or clear items. Balanced accuracy is the mean of recall on smoke frames and specificity on clear frames. The detector reads the same reference frame, so its balanced accuracy sits under Grounded , and its slopes are those of its recall and false-positive rate. Intervals resample the 17 fires (20,000 resamples) and widen percentile widths by 11.5 percent as in Table 3 ;

\blacktriangle

marks a per-model interval above zero. The 20-minute pooled interval is degenerate because its one new and one removed false positive fall on the same frame. No interval is adjusted for multiplicity. The mint rows’ slopes are level slopes; Detector, matched cuts the detector to each model’s own grounded false-positive count and pools as for the models.

Model	Bare	Grounded	Change	Gained/lost	FP $+$ / $-$	Bal. acc. change	First design
	Recall on 84 smoke frames				84 clear	Both labels
claude-opus-4.8	0.536	0.583	+0.048 ▲ [0.004, 0.117]	4/0	0/0	+0.024 [0.002, 0.058]	+0.071
claude-opus-5	0.607	0.667	+0.060 ▲ [0.017, 0.112]	5/0	5/0	0.000 [ $-0.069$ , 0.048]	+0.036
gemini-3.1-pro	0.595	0.726	+0.131 ▲ [0.063, 0.220]	11/0	1/0	+0.060 [0.020, 0.104]	+0.179
gpt-6-astra	0.655	0.774	+0.119 ▲ [0.046, 0.227]	10/0	2/2	+0.060 [0.000, 0.134]	+0.107
Qwen3-VL	0.571	0.607	+0.036 [ $-0.004$ , 0.072]	3/0	1/0	+0.012 [ $-0.001$ , 0.024]	+0.060

Appendix

Table A31: With matched gaps and masked timestamps, the reference still raises smoke recall for the four proprietary models. Smoke detection rerun on 168 FIgLib targets from 28 sequences of 17 fires. The 84 clear frames lie 30, 20, and 10 minutes before the first visible plume, and the 84 smoke frames 5, 15, and 25 minutes after it. Each target’s reference is the frame nearest 10, 20, or 30 minutes earlier, always before the plume, so each label meets each gap once per sequence. Both arms see every frame with its printed capture time masked, and the prompt calls the reference “an earlier frame from the same camera”. One run per model and arm. Gained/lost : smoke frames the reference turns from missed to found and back. FP

+

-

: clear frames it turns into false positives and back. First design : the recall change of Table A30 ’s run on the same 84 smoke frames. Intervals resample the 17 fires (20,000 resamples) and widen percentile widths by 11.5 percent;

\blacktriangle

marks a per-model recall interval above zero.

Model	Bare acc.	Grounded acc.	Grd. $-$ Bare	Gained / lost	Closed-form 48	Other 360
Reference points
Per-question majority	0.627	0.627			0.688	0.619
Majority + closed-form		0.664			1.000	0.619
Closed-form rule					1.000
Proprietary models
claude-opus-4.8	0.642	0.642	+0.000	17 / 17	0.667 $\to$ 0.896	0.639 $\to$ 0.608

Appendix

Table A32: A trivial hybrid rule outscores every model by point estimate, five of six sit within paired error of the held-out majority comparator, and Llama 4 Maverick sits below it; the thermal block’s point estimates rise on the closed-form questions and fall on the remaining items. Aerial question answering on WildFireVQA, 408 items over 390 FLAME 3 frame clusters, scored by exact match. Point estimates are followed by 95 percent frame-clustered bootstrap intervals. Gained / lost counts item-level flips. Closed-form 48 and Other 360 report accuracy bare

\to

grounded on the respective question subsets.

Core set
	48 closed-form items		347 remaining items
Model	Gained / lost	$p$	Bare $\to$ grounded	Gained / lost	$p$
claude-opus-4.8	11 / 0	0.001	0.640 $\to$ 0.608	6 / 17	0.035
claude-opus-5	13 / 0	$<$ 0.001	0.654 $\to$ 0.622	4 / 15	0.019
gemini-3.1-pro	13 / 0	$<$ 0.001	0.628 $\to$ 0.608	9 / 16	0.230
gpt-6-astra	7 / 0	0.016	0.617 $\to$ 0.608	7 / 10	0.629

Appendix

Table A33: On the 48 closed-form aerial items the thermal summary block raises accuracy for 14 of the sixteen full-capability models, eight at unadjusted item-level

p<0.05

, with no decline at that threshold; on the 347 remaining items, accuracy falls for all six core models and only the two Claude models meet that threshold. Exact two-sided McNemar paired tests on the prespecified aerial item partition. Gained / lost counts item-level flips from bare to grounded. Values of

p<0.05

indicate unadjusted two-sided significance. The 347-item columns report the core six only, so the sweep rows leave them blank. On the 13 no-fire-branch items, whose labels a maximum below 200 ∘ C fixes, no core model shows a resolved change. A PD1 rule of the same form would move 3 more items out of the 347 without changing any gained, lost, or

p

entry. Llama 4 Scout moves from 33 to 31 of the 48 and Ministral 3 8B from 34 to 30, with no recorded error in either arm.

Proprietary models
			Label source			Applicability
		All	Formula, telemetry, or detector	Same, without closed form	Model- verified	$=1.0$	$<1.0$
Model	Arm	408 (34)	132 (11)	84 (7)	276 (23)	284 (33)	124 (25)
Held-out majority		0.627	0.614	0.571	0.634	0.637	0.605
claude-opus-4.8	bare	0.642	0.500	0.405	0.710 ▲	0.651	0.621
claude-opus-4.8	grounded	0.642	0.583	0.405	0.670	0.676	0.565

Appendix

Table A34: On the 132 items labeled by formula, telemetry, or detector, every arm scores below the held-out majority; on the 276 model-verified items, two bare arms have unadjusted intervals above it. Accuracy on WildFireVQA per core model and arm, pooled and within five strata, beside the held-out per-question majority scored on the same items. Column heads give items and, in parentheses, question types. Each question type has one label source, so the label-source columns compare disjoint sets of question types and cannot separate label quality from question type. The 84-item column removes the 48 closed-form items, all of which carry formula labels. Markers flag an unadjusted paired 95 percent frame-clustered interval wholly above ( ▲ ) or below ( ▼ ) zero: model minus majority, and grounded minus bare in the last block. Because differences move in whole items, five unmarked cells have a frame-clustered interval ending at exactly zero and could gain a marker under another seed. With question types resampled, only claude-opus-5 bare on model-verified items keeps a comparator marker, and five arm-contrast markers remain. Resampling both dimensions leaves none. Unadjusted exact McNemar tests reach

p<0.05

on every comparator marker and on six of the ten arm-contrast markers. Holm adjustment over the twelve arms of each column leaves no comparator

p

-value below 0.05. Intervals use 20,000 resamples and base seed 20260915.

Model	Alloc.	Smoke	Aerial	Danger	Tool
claude-opus-4.8	0/0	0/0	0/0	0/0	0
claude-opus-5	0/0	0/0	0/0	0/0	0
gemini-3.1-pro	431/355	n/r/296	266/197	484/572	1,158
gpt-6-astra	33/31	0/0	30/21	55/45	21

Appendix

Table A35: Only Gemini and GPT models report nonzero reasoning tokens. Median reasoning tokens per response (bare/grounded; bare arm for tool use). n/r: not reported on every response (22 of 224 gemini-3.1-pro bare smoke records carry the field).

Explore similar work

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Sep 7, 2026Pengfei Li, Naufal Suryanto, Sicheng Zhang +2LLM Safety Benchmarks LLM Safety Evaluation
Khalifa University · University of Western Australia
Does Your Wildfire Prediction Model Actually Work, or Just Score Well?
May 14, 2026Yangshuang Xu, Yuyang Dai, Liling Chang +2Geospatial Foundation Models Wildfire Forecasting
Florida State University · Northeastern University
Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5
Jul 8, 2026Yongcan Huang, Li Jiang, Ze Yu LiuWildfire Forecasting OOD Generalization
College of Engineering, University of Georgia, Athens, Georgia, USA · Information System Department, UMBC, Baltimore, Maryland, USA · College of Graduate and Professional Studies, Trine University, Angola, Indiana, USA +1

Sep 7, 2026, cs.CVJ/K move · Enter open · S save

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid+1

Khalifa University · University of Western Australia

LLM Safety Benchmarks LLM Safety Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.

Abstract

Explore similar work

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5