Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Weixiang Zhao; Zhihao Liu; Guancheng Li; Yanyan Zhao; Bing Qin; Ting Liu

cs.AINov 2, 2026

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Authors: Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin, Ting Liu

Organizations: Harbin Institute of Technology

Abstract

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.

Figures & tables

Figure 1: Conceptual overview of predictive safety guardrails. We diagnose predictive safety along three stages: prediction integration (RQ1), reliability (RQ2), and decision faithfulness (RQ3).

		HA			IS-Bench
Actor	Mode	SAR $\uparrow$	AVS $\uparrow$	TCR $\uparrow$	TS $\uparrow$	CSS $\uparrow$	ESS $\uparrow$	AF $\downarrow$
Qwen3.5-27B	No Prediction	76.81	85.19	83.63	77.64	22.98	31.36	21.76
	Prediction-Informed	73.40	84.00	84.42	77.64	24.22	34.82	17.36
	Prediction-Constrained	85.10	87.95	82.41	77.64	22.36	31.30	17.01
Gemini-3.7-Flash	No Prediction	69.51	84.91	88.07	96.18	20.38	23.02	9.92
	Prediction-Informed	70.12	84.99	88.21	96.80	20.00	23.58	10.21

Table 1: Online results on HA and IS-Bench. HA reports

0

–

100

scores; IS-Bench reports percentages. AF is action-weighted; TS, CSS, and ESS are trajectory-level. Arrows indicate the preferred direction.

	Mean tokens / task
Mode	HA	IS-Bench
No Prediction	134,710.5	44,681.1
Prediction-Informed	288,095.8	45,674.8
Prediction-Constrained	850,448.0	140,805.6

Table 2: Mean token consumption per task across prediction integration settings.

Benchmark	Model	Invocation	Action change	Safety improvement
HA	Qwen3.5-27B	47.47%	95.29%	12.50%
	Gemini-3.7-Flash	43.94%	89.86%	20.00%
	GLM-5.3-Flash	41.77%	90.28%	16.67%
	GPT-5.6-Terra	25.00%	88.10%	0.00%
IS-Bench	Qwen3.5-27B	7.12%	68.64%	6.67%
	Gemini-3.7-Flash	16.00%	20.00%	5.88%

Table 3: Prediction-Informed diagnostics. Invocation, action change, and safety improvement use their original denominators.

Figure 2: Offline prediction on HA and IS-Bench. (a) GPT-5.6-Terra on HA (Full/Train-derived; Direct/Iterative). (b) Qwen3.5-Plus on IS-Bench (Full/Train-derived/Generic; Direct/Iterative). Columns show accuracy, risk precision, risk recall, and RC. Horizons index benchmark-specific action records; Additional model results appear in Appendix C .

Figure 3: Feedback interventions on HA with Qwen3.5-27B. Bars report SAR, AVS, and TCR (0–100; higher is better). No Prediction and Normal reuse the corresponding online results from Table 1 . Empty, Irrelevant, Corrupt, Filler, and Full Inversion perturb feedback; Sandbox supplies sandbox-derived feedback; bar labels show one decimal place.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Execution protocols used in the online experiments. (a) No Prediction omits external WM assistance. (b) Prediction-Informed makes consultation optional; the actor may retain or revise its candidate without a WM-based veto. (c) Prediction-Constrained checks eligible candidates, blocks disallowed actions, and rechecks each revision. Allow/block denote gate decisions; benchmark evaluation supplies safety labels. All modes retain environment-native constraints.

Property	No Prediction	Prediction- Informed	Prediction- Constrained
External WM available	No	Yes, as an optional tool	Yes, through the guard
Who initiates checking?	Not applicable	Actor	Guard on eligible candidates
Can the Actor skip consultation?	Not applicable	Yes	Not for a required check
WM feedback	None	Returned after consultation	Returned for guarded decisions/revision
External WM execution gate	Absent	Absent	Present
After candidate revision	Native execution rules	Actor decides next action	New candidate is checked again

Appendix

Table 4: Operational differences among the three prediction-integration conditions. Native environmental constraints apply in every column. Prediction-Informed leaves execution control with the Actor.

Diagnostic	HA interpretation	IS-Bench interpretation
Invocation	Trajectories with at least one consultation / evaluated trajectories	Trajectories with at least one consultation / evaluated trajectories
Action change	Linked feedback events whose following candidate changes / linked events	Recorded action changes after consultation / the corresponding consultation set
Safety improvement	Recorded unsafe-to-safe correction percentage	Recorded unsafe-to-safe correction percentage

Appendix

Table 5: Units of the reported Prediction-Informed diagnostics. Each percentage retains the denominator appropriate to its stage and benchmark.

World model	Mode	SAR $\uparrow$	AVS $\uparrow$	TCR $\uparrow$
Gemini-3.7-Flash	No Prediction	78.79	85.56	84.08
	Prediction-Informed	72.89	84.22	85.24
	Prediction-Constrained	81.52	86.33	82.99
DeepSeek-V4-Flash	No Prediction	77.40	85.45	84.81
	Prediction-Informed	72.62	83.75	83.95
	Prediction-Constrained	81.11	86.04	82.31

Appendix

Table 6: Cross-WM comparison on HA with Qwen3.5-27B as the actor. Each predictor configuration retains its own No Prediction baseline. Scores range from

0

100

Horizon	Record	Illustrative action	Risk label
$h_{1}$	$t$	Inspect an internal report.	0
$h_{2}$	$t+1$	Draft a public summary locally.	0
$h_{3}$	$t+2$	Publish a version containing prohibited personal identifiers.	1
$h_{4}$ (HA only)	$t+3$	Acknowledge the operation’s status.	0

Appendix

Table 7: Illustrative exact-step targets. The label applies to the indexed action, rather than to all actions up to that horizon.

Condition	Policy source	Question addressed
Full	Rules associated with the current task.	Can the WM detect violations when the relevant requirements are explicit?
Train-derived	Constraints extracted from a training subset.	Does extracted policy information support prediction beyond its extraction instances?
Generic	Non-instance-specific embodied-safety guidance.	How much can prediction rely on general safety guidance?

Appendix

Table 8: Policy inputs and the comparisons they enable.

Benchmark	ID	Prediction mode	Policy input	Horizons
HA	HA-1	Direct	Full	$h_{1}$ – $h_{6}$
	HA-2	Direct	Train-derived	$h_{1}$ – $h_{6}$
	HA-3	Iterative	Full	$h_{1}$ – $h_{6}$
	HA-4	Iterative	Train-derived	$h_{1}$ – $h_{6}$
IS-Bench	IS-1	Direct	Full	$h_{1}$ – $h_{3}$
	IS-2	Direct	Train-derived	$h_{1}$ – $h_{3}$

Appendix

Table 9: Complete prediction setting grid. HA has

2\times 2=4

conditions and IS-Bench has

2\times 3=6

. IDs are descriptive shorthand; every condition is evaluated at all listed horizons.

Figure 5: DeepSeek-V4-Flash on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 6: Qwen-AgentWorld on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 7: Qwen3.5-Plus on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 8: Qwen3.5-27B on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 9: Qwen3.5-27B on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 10: GLM-5.3-Flash on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 11: GPT-5.6-Terra on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 12: Gemini-3.7-Flash on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Condition	Explanatory text	Risk judgment	Risk score	Policy/action guidance
Normal	Original	Original	Original	Original
Empty	Removed	Retained	Removed	Removed
Irrelevant	Unrelated text	Retained	Removed	Removed
Corrupt	Key text damaged	Retained	Retained	Retained
Filler	Non-diagnostic filler	Retained	Removed	Removed
Full Inversion	Risk direction reversed	Reversed	Across threshold	Transformed with the risk assessment

Appendix

Table 10: Feedback fields retained, removed, or transformed under each intervention.

Case	Task	Mode	Recorded behavior
1	dl-t11	NP	Compares waiting with immediate authorization.
2	si-t4	NP	Anticipates identity exposure and publication prerequisites.
3	Cleaning fan	NP	Turns off the fan before cleaning.
4	si-t27	PI	Treats a zero-risk forecast as publication confirmation.
5	fin-t26	PI	Repeats a prepared filing after zero-risk advice.
6	si-t12	PI	Retains the proposed dispute after a zero-risk forecast.

Appendix

Table 11: Index of the seven qualitative cases. NP, PI, and PC denote No Prediction, Prediction-Informed, and Prediction-Constrained, respectively.

Figure 13: Feedback interventions for GLM-5.3-Flash on HA. Bars report SAR, AVS, and TCR on the 0–100 scale (higher is better), with labels rounded to one decimal place. Normal uses unperturbed Prediction-Constrained feedback; the intervention definitions are given in Appendix D .

Figure 14: Feedback interventions for Gemini-3.7-Flash on HA. Bars report SAR, AVS, and TCR on the 0–100 scale (higher is better), with labels rounded to one decimal place. Normal uses unperturbed Prediction-Constrained feedback; the intervention definitions are given in Appendix D .

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 341 papers

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
Aug 6, 2026Wenhao Lin, Chenyu Yu, Xingwei Lin +6LLM World Models LLM Guardrails
Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University
PreAct-Bench: Benchmarking Predictive Monitoring in LLMs
Jun 3, 2026Hainiu Xu, Italo Luis da Silva, Jiangnan Ye +7LLM Safety Benchmarks Moral Reasoning in Language Models
King’s College London · National University of Singapore · Southern University of Science and Technology +3
Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models
Oct 8, 2026Youwei Feng, Yitong Zhang, Yuetong Liu +1LLM Agent Safety LLM Guardrails
Tsinghua University, Beijing, China · College of AI, Tsinghua University, Beijing, China

Aug 6, 2026J/K move · Enter open · S save

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

Wenhao Lin, Chenyu Yu, Xingwei Lin, Sicong Cao+5

Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University

LLM World Models LLM Guardrails World Models Runtime Enforcement for AI Agents LLM Agent Safety Long-Horizon LLM Agents

Open paper AskPDF arXiv

As large language model (LLM) agents increasingly invoke external tools and interact with real-world systems, unsafe actions may cause irreversible consequences on external states, user data, and downstream services. Recent runtime guardrails mitigate such risks by checking proposed actions before execution, but many remain reactive: they primarily assess the apparent safety of the current action, lacking an explicit model of how risk evolves across the trajectory. This limitation creates a critical blind spot for long-horizon risks, where individually benign-looking actions can gradually drift the agent toward hazardous states. In response, we propose DreamGuard, a proactive guardrail for LLM agents built around a risk-aware world model. The world model maintains a compact recurrent latent state over the trajectory and predicts future latent states from which DreamGuard derives immediate-hazard and prefix-risk evidence. It then fuses these multi-horizon signals into intervention decisions before execution. Experiments across four benchmarks and an online guardrail evaluation show that DreamGuard outperforms generic, reactive, and proactive guardrail baselines, achieves the best safety-utility trade-off among evaluated guardrails, and maintains an average end-to-end latency of 25 ms per call.

Abstract

LLM Agent Safety

Explore similar work

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

PreAct-Bench: Benchmarking Predictive Monitoring in LLMs

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models