HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

Wangshu Zhu; Xueqi Cheng; Liang Wu; Yushun Dong

cs.AIOct 2, 2026

HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

Authors: Wangshu Zhu, Xueqi Cheng, Liang Wu, Yushun Dong

Organizations: Department of Computer Science, Florida State University · Nokia

Abstract

Understanding and assessing natural hazards is essential for disaster preparedness and risk reduction. Recent advances in large language models have spurred growing interest in AI agents for hazard analysis, particularly their ability to integrate scientific data, models, and tools into automated workflows. However, effective automation requires agents to determine which scientific methods are appropriate for a given event and executable with the available data and tools. As new evidence and execution results become available, these conditions can change, requiring agents to reconsider their choices. We formulate this problem as state-dependent scientific route selection and introduce HazardWeaver. Specifically, HazardWeaver first leverages the Hazard Knowledge Compiler to extract evidence-linked conditions governing scientific applicability, then its Hazard Capability Graph represents executable scientific capabilities and checks compatibility between their inputs and outputs. Using these complementary representations, the Hazard Weaver Agent component selects applicable and executable routes, carries out their workflows, and revises its decisions as the analysis state changes. To evaluate both the scientific outputs and the decisions that produce them, we introduce the Hazard Weaver Benchmark, comprising 141 instances across seven single-hazard domains and four multi-hazard interaction classes. The benchmark accommodates multiple valid scientific routes and evaluates output correctness, route validity, and justified abstention. Extensive experiments on this benchmark show that HazardWeaver outperforms existing agent systems, with the largest gains on tasks with multiple eligible scientific routes. Our code is publicly available at https://github.com/LabRAI/HazardWeaver.

Figures & tables

Figure 2: Overview of HazardWeaver. HKC and HCG identify scientifically applicable and currently executable routes, whose intersection defines the eligible route set for HWA. HWA selects, executes, and revises model-backed routes as the analysis state changes, while HWB independently evaluates the resulting artifact, route validity, and recovery or abstention.

Method	Single-hazard	Multi-hazard	One route	Multiple routes	Overall DCA
AIDE	$22.2^{\scriptscriptstyle+9.7}_{\scriptscriptstyle-7.3}$	$23.5^{\scriptscriptstyle+13.3}_{\scriptscriptstyle-9.5}$	$28.9^{\scriptscriptstyle+14.5}_{\scriptscriptstyle-11.2}$	$19.8^{\scriptscriptstyle+9.1}_{\scriptscriptstyle-6.7}$	$22.7^{\scriptscriptstyle+7.6}_{\scriptscriptstyle-6.1}$
AutoML-Agent	$26.7^{\scriptscriptstyle+9.9}_{\scriptscriptstyle-8.1}$	$27.4^{\scriptscriptstyle+13.6}_{\scriptscriptstyle-10.3}$	$24.4^{\scriptscriptstyle+14.3}_{\scriptscriptstyle-10.2}$	$28.1^{\scriptscriptstyle+9.7}_{\scriptscriptstyle-8.0}$	$26.9^{\scriptscriptstyle+7.9}_{\scriptscriptstyle-6.6}$
ReAct	$28.9^{\scriptscriptstyle+10.1}_{\scriptscriptstyle-8.4}$	$31.4^{\scriptscriptstyle+13.6}_{\scriptscriptstyle-11.1}$	$31.1^{\scriptscriptstyle+14.6}_{\scriptscriptstyle-11.6}$	$29.2^{\scriptscriptstyle+9.7}_{\scriptscriptstyle-8.2}$	$29.8^{\scriptscriptstyle+8.0}_{\scriptscriptstyle-6.9}$
Reflexion	$30.0^{\scriptscriptstyle+10.1}_{\scriptscriptstyle-8.5}$	$35.3^{\scriptscriptstyle+13.7}_{\scriptscriptstyle-11.7}$	$51.1^{\scriptscriptstyle+13.9}_{\scriptscriptstyle-14.1}$	$22.9^{\scriptscriptstyle+9.4}_{\scriptscriptstyle-7.2}$	$31.9^{\scriptscriptstyle+8.1}_{\scriptscriptstyle-7.1}$
MLE-STAR	$35.6^{\scriptscriptstyle+10.3}_{\scriptscriptstyle-9.2}$	$29.4^{\scriptscriptstyle+13.6}_{\scriptscriptstyle-10.7}$	$35.6^{\scriptscriptstyle+14.6}_{\scriptscriptstyle-12.4}$	$32.3^{\scriptscriptstyle+9.9}_{\scriptscriptstyle-8.5}$	$33.3^{\scriptscriptstyle+8.2}_{\scriptscriptstyle-7.2}$
DS-Agent	$33.3^{\scriptscriptstyle+10.3}_{\scriptscriptstyle-8.9}$	$33.3^{\scriptscriptstyle+13.7}_{\scriptscriptstyle-11.3}$	$40.0^{\scriptscriptstyle+14.5}_{\scriptscriptstyle-13.0}$	$30.2^{\scriptscriptstyle+9.8}_{\scriptscriptstyle-8.3}$	$33.3^{\scriptscriptstyle+8.2}_{\scriptscriptstyle-7.2}$

Table 1: Comparison with agent baselines. Each cell reports DCA (%) with the upper and lower offsets of its Wilson 95% confidence interval. Methods are ordered by overall DCA.

Figure 3: Scientific-constraint ablation results. DCA (%) on the 96 multiple-route instances. Shaded bands indicate Wilson 95% interval.

Backbone	Fixed	Updated
Mixtral	$5.0^{\scriptscriptstyle+11.5}_{\scriptscriptstyle-3.6}$	$15.0^{\scriptscriptstyle+14.1}_{\scriptscriptstyle-7.9}$
OLMo	$17.5^{\scriptscriptstyle+14.4}_{\scriptscriptstyle-8.7}$	$25.0^{\scriptscriptstyle+15.2}_{\scriptscriptstyle-10.8}$
DeepSeek	$22.5^{\scriptscriptstyle+15.0}_{\scriptscriptstyle-10.2}$	$35.0^{\scriptscriptstyle+15.5}_{\scriptscriptstyle-12.9}$
Gemma	$20.0^{\scriptscriptstyle+14.8}_{\scriptscriptstyle-9.5}$	$42.5^{\scriptscriptstyle+15.3}_{\scriptscriptstyle-14.0}$
Llama	$15.0^{\scriptscriptstyle+14.1}_{\scriptscriptstyle-7.9}$	$90.0^{\scriptscriptstyle+6.0}_{\scriptscriptstyle-13.0}$
Pooled	16.0	41.5

Table 2: Route update ablation. DCA (%) on 40 applicable instances.

Backbone	DCA	Macro DCA	Artifact validity ( $E_{q}$ )	Trajectory validity ( $V_{q}$ )
OLMo	60.3 [52.0, 68.0]	61.9 [54.4, 71.2]	78.0 [70.5, 84.1]	91.5 [85.7, 95.1]
Mixtral	70.2 [62.2, 77.1]	70.8 [57.6, 83.9]	79.4 [72.0, 85.3]	92.2 [86.6, 95.6]
Gemma	79.4 [72.0, 85.3]	78.6 [65.2, 91.5]	86.5 [79.9, 91.2]	92.9 [87.4, 96.1]
DeepSeek	86.5 [79.9, 91.2]	86.6 [77.2, 94.3]	90.8 [84.9, 94.5]	92.2 [86.6, 95.6]
Llama	89.4 [83.2, 93.5]	89.5 [79.7, 97.0]	92.2 [86.6, 95.6]	95.0 [90.1, 97.6]

Table 3: Cross-backbone performance of Full HazardWeaver. DCA is the headline metric, and Macro DCA assigns equal weight to the 11 tracks.

E_{q}

and

V_{q}

report the corresponding marginal artifact- and trajectory-validity rates under the same sealed evaluator.

Panel of Figure 4 — Figure 4: Cross-backbone behavior of Full HazardWeaver. Panels report DCA by (a) hazard composition, (b) route multiplicity, and (c) benchmark track.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: HWB inventory. Instance counts across the 7 single- and 4 multi-hazard tracks.

Hazard task	Scientific source and role
Wildfire spread	WildfireSpreadTS. Time-indexed fire observations for evaluating the spatial and temporal progression of active-fire spread.
Flood inundation	FloodCastBench. Hydrological forcing fields and reference flood extents for dynamic inundation analysis.
Rainfall-triggered landslide	NASA LHASA v2. Global landslide nowcasts and reference occurrences for rainfall-triggered susceptibility analysis.
Earthquake analysis	SeisBench. Waveforms and reference labels for phase picking, event association, and seismic-source estimation.
Tropical-cyclone tracking	TCBench. Cyclone trajectories and lead-time-specific references for evaluating forecast track error.
Drought outlook	NOAA CPC drought products. Standardized precipitation indicators and operational references for drought assessment.

Appendix

Table A.1: Scientific data sources used in HWB construction. Each source supplies observed events, forcing variables, reference products, or evaluator targets; the corresponding publications and official records are cited in Section 4.

Figure A.2: Joint benchmark strata. Instance counts by hazard and route composition.

Field group	Contents
Task identity	Hazard domain, task specification, requested output, and route family.
Scientific conditions	Assumptions, required observations, validity ranges, calibration regime, scale, and uncertainty treatment.
Capability requirements	Required datasets, models, formulas, transformations, and compatibility or dependency constraints.
Output contract	Variable definition, units, format, spatial and temporal support, resolution, and artifact schema.
Evaluation	Task-native metric, tolerance, and acceptance threshold.
Provenance	Scientific source, capability version, supporting evidence span, and benchmark-construction record.

Appendix

Table B.1: Principal Route Card fields.

Operation	Purpose and state effect
Inspect state	Reads evidence, artifacts, and capability status; no state change.
Inspect route	Reads route conditions, capabilities, and contract; no state change.
Execute capability	Runs a capability; records artifact and provenance, or the failure.
Request information	Obtains a permitted unresolved observation; adds it as evidence.
Repair workflow	Inserts a transformation or subpath; keeps the route-level target.
Change route	Commits to another route from the recomputed eligible set.

Appendix

Table B.2: Grounded tools available to HWA. Operations are instantiated only when permitted by the task and remaining execution state.

Hazard track	Scientific outputs and evaluation
Active-fire spread	Fire probability, occupancy, or perimeter, evaluated using average precision, intersection over union, and registered field checks.
Dynamic flood inundation	Inundation depth and extent, evaluated using depth RMSE over the scenario-specific spatial support.
Rainfall-triggered landslide	Landslide occurrence probability, evaluated using area under the precision–recall curve.
Seismic phase, source, and motion	Phase picks, event associations, source parameters, and ground motion, evaluated using pick F1, association score, and coverage.
Tropical-cyclone trajectory	Forecast storm trajectories, evaluated using lead-time-specific track error in kilometers.
Drought outlook	Drought categories or probabilities, evaluated using the registered operational outlook-skill criterion.

Appendix

Table C.1: Task-native evaluator families. DCA applies the artifact criteria below jointly with route and execution validity.

Backbone	Model identifier
OLMo-3.1-32B	allenai/Olmo-3.1-32B-Instruct
Mixtral-8x22B	mistralai/Mixtral-8x22B-Instruct-v0.1
Gemma-4-31B-it	google/gemma-4-31B-it
Llama-3.3-70B	meta-llama/Llama-3.3-70B-Instruct
DeepSeek-V4.1-Flash	deepseek-v4.1-flash

Appendix

Table D.1: Policy backbones. Model identifiers used with Full HazardWeaver; all non-policy components remain fixed.

Method	Adapted principle and HWB realization
AIDE	Iterative candidate refinement. Refines a proposed scientific route for up to three rounds using the task and Route Cards, then commits directly.
DS-Agent	Case-based retrieval. Retrieves the three most similar training-side route cases and uses them to rank the registered candidates.
R&D-Agent	Hypothesis-driven ordering. Forms a task hypothesis, orders the candidate scientific routes against it, and commits to the highest-ranked route.
MLE-STAR	Metric-directed program search. Infers the required scientific metric and output program before ranking candidate routes against those requirements.
AutoML-Agent	Pipeline and schema selection. Infers the task input–output schema and ranks routes by their ability to realize a compatible scientific pipeline.
DisasterBench-ToT	Deliberate search over disaster tools. Treats registered capabilities as typed disaster tools and performs bounded tree search over candidate route choices.

Appendix

Table E.1: Baseline adaptations used in RQ1. “Rounds” refers to route-selection or reflection rounds before the terminal commitment.

Backbone	HKC off	HCG untyped	Full
OLMo	38 (39.6)	39 (40.6)	49 (51.0)
Mixtral	34 (35.4)	32 (33.3)	65 (67.7)
Gemma	50 (52.1)	55 (57.3)	76 (79.2)
DeepSeek	68 (70.8)	71 (74.0)	86 (89.6)
Llama	44 (45.8)	46 (47.9)	90 (93.8)

Appendix

Table F.1: Scientific-constraint ablations. DCA on the 96 route-choice instances; entries give count and percentage.

Figure F.1: Paired significance tests. Discordant DCA counts between HazardWeaver and each baseline; annotations give McNemar

p

-values.

Figure F.2: Joint-stratum performance. DCA across hazard composition and route multiplicity; annotations report successful instances over stratum size.

Figure F.3: Track-level DCA for Full HazardWeaver. Each entry reports track-level DCA (%).

Information	Agent	Eval.
Task and initial state	✓	✓
Candidate routes	✓	✓
Capability interface	✓	✓
Public output contract	✓	✓
Reference artifact		✓
Sealed route criterion		✓

Appendix

Table G.1: Information separation. Availability for execution and scoring.

Artifact	Contents
Sealed inventory	The 141 instance identifiers, track assignments, task specifications, initial states, route strata, and evaluator bindings.
Route Cards	Scientific conditions, required observations, capability signatures, output contracts, uncertainty fields, and evidence provenance.
Capability registry	Executable entry points, versioned dependencies, typed input and output ports, transformations, and validation metadata.
Evaluator contracts	Task-native metrics, tolerances, route criteria, cutoff rules, provenance requirements, and abstention conditions.
Run configurations	Model identifiers, stochastic settings, execution protocol, and method-specific selector configuration.
Per-instance outputs	Selected routes, state transitions, capability invocations, generated artifacts, provenance records, and terminal evaluation reports.

Appendix

Table I.1: Reproducibility artifacts accompanying HWB.

LLM Agent Planning

Large language model (LLM) agent planning studies how agents decompose goals, choose actions and tools, and adapt plans from feedback.

See all 225 papers

Jul 21, 2026

DeepSafe-SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

Large language models (LLMs) increasingly support science, but they can also convert hazardous scientific knowledge into actionable misuse guidance. Existing benchmarks often rely on templated queries disconnected from real-world hazards, and employ LLM-as-a-Judge paradigms without domain grounding. To address this, we introduce SciHazard, a real-world-grounded benchmark for scientific risks and a dataset agnostic evaluation framework for measuring harmfulness. SciHazard contains 2400 hazardous questions and 600 oversafety questions across 12 disciplines, with both queries grounded in regulated entities and documented failure scenarios. To compute DeHarm-Score , we develop a decomposed evaluating procedure that combines query hazard severity, refusal behavior, and response-level risk. For non-refused responses, it further decomposes response-level harm into Executability, quantified via dynamic checklists with importance weighting, and Net-new risk, assessed through retrieval-augmented claim extraction and synthesis-barrier verification. An expert-validation study shows that DeHarm-Score improves agreement with expert annotations by 90.17% over the strongest baseline. We benchmark 31 frontier LLMs and deep research agents in an extensive scientific safety evaluation. Notably, deep research agents yield 32.3% higher mean DeHarm-Score than standard LLMs, exposing autonomous agents as a critical blind spot in current safety defenses. Code and dataset are available at https://github.com/AI45Lab/DeepSafe-Sci.

Chunxiao Li, Yuan Xiong, Lijun Li +4

Shanghai Artificial Intelligence Laboratory

LLM Safety Benchmarks AI Agent Safety Benchmarks

Ask PDF

Jun 6, 2026

SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents

LLM-based scientific agents have shown strong capacity for autonomous research, yet their safety layers remain structurally divorced from core reasoning: they inspect pipeline outputs rather than shaping the deliberation that produces them. This separation opens two failure modes: safety signals accumulated at one stage are discarded before the next, and sequences of individually benign tool calls can compose into harmful outcomes that no single-step filter detects. To address these challenges, we introduce SciTrace, a framework that weaves safety reasoning into every stage of the scientific agent pipeline. SciTrace couples two complementary mechanisms: a Safety-Intrinsic Reasoning Loop (SIR) that maintains a cumulative risk state across the Thinker, Experimenter, Writer, and Reviewer stages through joint task-and-safety deliberation, and a Compositional Tool-Chain Verifier (CTV) that performs trajectory-aware safety checks before execution, catching risks that surface only across multi-step tool sequences. Evaluated on 240 high-risk research tasks and 120 tool-related risk tasks spanning six scientific domains, SciTrace achieves state-of-the-art (SOTA) safety among compared frameworks across four backbone models: it consistently improves tool call safety and adversarial robustness while preserving scientific output quality, and it uncovers 78.8% of the compositional tool-chain escapes that single-step monitors miss. The project website is available at https://opensciagent.github.io/SciTrace/.

Tanush Swaminathan, Runmin Jiang, Letian Zhang +1

Carnegie Mellon University · Allen Institute

AI Agents for Scientific Discovery LLM Safety Evaluation LLM Agent Safety

Ask PDF

Apr 3, 2026

AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents

Computer-use agents extend language models from text generation to persistent action over tools, files, and execution environments. Unlike chat systems, they maintain state across interactions and translate intermediate outputs into concrete actions. This creates a distinct safety challenge in that harmful behavior may emerge through sequences of individually plausible steps, including intermediate actions that appear locally acceptable but collectively lead to unauthorized actions. We present AgentHazard, a benchmark for evaluating harmful behavior in computer-use agents. AgentHazard contains 2,653 instances spanning diverse risk categories and attack strategies. Each instance pairs a harmful objective with a sequence of operational steps that are locally legitimate but jointly induce unsafe behavior. The benchmark evaluates whether agents can recognize and interrupt harm arising from accumulated context, repeated tool use, intermediate actions, and dependencies across steps. We evaluate AgentHazard on Claude Code, OpenClaw, and IFlow using mostly open or openly deployable models from the Qwen3, Kimi, GLM, and DeepSeek families. Our experimental results indicate that current systems remain highly vulnerable. In particular, when powered by Qwen3-Coder, Claude Code exhibits an attack success rate of 73.63%, suggesting that model alignment alone does not reliably guarantee the safety of autonomous agents.

Yunhao Feng, Yifan Ding, Yingshui Tan +6

Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2

LLM Safety Benchmarks Computer-Use Agent Benchmarks Computer-Use Agent Safety AI Agent Safety AI Agent Safety Benchmarks LLM Agent Safety

Ask PDF

Abstract

LLM Agent Planning

Explore similar work

DeepSafe-SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents

AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents