Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Weixiang Zhao; Zhihao Liu; Guancheng Li; Yanyan Zhao; Bing Qin; Ting Liu

cs.AINov 2, 2026

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Authors: Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin, Ting Liu

Organizations: Harbin Institute of Technology

Abstract

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.

Figures & tables

Figure 1: Conceptual overview of predictive safety guardrails. We diagnose predictive safety along three stages: prediction integration (RQ1), reliability (RQ2), and decision faithfulness (RQ3).

		HA			IS-Bench
Actor	Mode	SAR $\uparrow$	AVS $\uparrow$	TCR $\uparrow$	TS $\uparrow$	CSS $\uparrow$	ESS $\uparrow$	AF $\downarrow$
Qwen3.5-27B	No Prediction	76.81	85.19	83.63	77.64	22.98	31.36	21.76
	Prediction-Informed	73.40	84.00	84.42	77.64	24.22	34.82	17.36
	Prediction-Constrained	85.10	87.95	82.41	77.64	22.36	31.30	17.01
Gemini-3.7-Flash	No Prediction	69.51	84.91	88.07	96.18	20.38	23.02	9.92
	Prediction-Informed	70.12	84.99	88.21	96.80	20.00	23.58	10.21

Table 1: Online results on HA and IS-Bench. HA reports

0

–

100

scores; IS-Bench reports percentages. AF is action-weighted; TS, CSS, and ESS are trajectory-level. Arrows indicate the preferred direction.

	Mean tokens / task
Mode	HA	IS-Bench
No Prediction	134,710.5	44,681.1
Prediction-Informed	288,095.8	45,674.8
Prediction-Constrained	850,448.0	140,805.6

Table 2: Mean token consumption per task across prediction integration settings.

Benchmark	Model	Invocation	Action change	Safety improvement
HA	Qwen3.5-27B	47.47%	95.29%	12.50%
	Gemini-3.7-Flash	43.94%	89.86%	20.00%
	GLM-5.3-Flash	41.77%	90.28%	16.67%
	GPT-5.6-Terra	25.00%	88.10%	0.00%
IS-Bench	Qwen3.5-27B	7.12%	68.64%	6.67%
	Gemini-3.7-Flash	16.00%	20.00%	5.88%

Table 3: Prediction-Informed diagnostics. Invocation, action change, and safety improvement use their original denominators.

Figure 2: Offline prediction on HA and IS-Bench. (a) GPT-5.6-Terra on HA (Full/Train-derived; Direct/Iterative). (b) Qwen3.5-Plus on IS-Bench (Full/Train-derived/Generic; Direct/Iterative). Columns show accuracy, risk precision, risk recall, and RC. Horizons index benchmark-specific action records; Additional model results appear in Appendix C .

Figure 3: Feedback interventions on HA with Qwen3.5-27B. Bars report SAR, AVS, and TCR (0–100; higher is better). No Prediction and Normal reuse the corresponding online results from Table 1 . Empty, Irrelevant, Corrupt, Filler, and Full Inversion perturb feedback; Sandbox supplies sandbox-derived feedback; bar labels show one decimal place.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Execution protocols used in the online experiments. (a) No Prediction omits external WM assistance. (b) Prediction-Informed makes consultation optional; the actor may retain or revise its candidate without a WM-based veto. (c) Prediction-Constrained checks eligible candidates, blocks disallowed actions, and rechecks each revision. Allow/block denote gate decisions; benchmark evaluation supplies safety labels. All modes retain environment-native constraints.

Property	No Prediction	Prediction- Informed	Prediction- Constrained
External WM available	No	Yes, as an optional tool	Yes, through the guard
Who initiates checking?	Not applicable	Actor	Guard on eligible candidates
Can the Actor skip consultation?	Not applicable	Yes	Not for a required check
WM feedback	None	Returned after consultation	Returned for guarded decisions/revision
External WM execution gate	Absent	Absent	Present
After candidate revision	Native execution rules	Actor decides next action	New candidate is checked again

Appendix

Table 4: Operational differences among the three prediction-integration conditions. Native environmental constraints apply in every column. Prediction-Informed leaves execution control with the Actor.

Diagnostic	HA interpretation	IS-Bench interpretation
Invocation	Trajectories with at least one consultation / evaluated trajectories	Trajectories with at least one consultation / evaluated trajectories
Action change	Linked feedback events whose following candidate changes / linked events	Recorded action changes after consultation / the corresponding consultation set
Safety improvement	Recorded unsafe-to-safe correction percentage	Recorded unsafe-to-safe correction percentage

Appendix

Table 5: Units of the reported Prediction-Informed diagnostics. Each percentage retains the denominator appropriate to its stage and benchmark.

World model	Mode	SAR $\uparrow$	AVS $\uparrow$	TCR $\uparrow$
Gemini-3.7-Flash	No Prediction	78.79	85.56	84.08
	Prediction-Informed	72.89	84.22	85.24
	Prediction-Constrained	81.52	86.33	82.99
DeepSeek-V4-Flash	No Prediction	77.40	85.45	84.81
	Prediction-Informed	72.62	83.75	83.95
	Prediction-Constrained	81.11	86.04	82.31

Appendix

Table 6: Cross-WM comparison on HA with Qwen3.5-27B as the actor. Each predictor configuration retains its own No Prediction baseline. Scores range from

0

100

Horizon	Record	Illustrative action	Risk label
$h_{1}$	$t$	Inspect an internal report.	0
$h_{2}$	$t+1$	Draft a public summary locally.	0
$h_{3}$	$t+2$	Publish a version containing prohibited personal identifiers.	1
$h_{4}$ (HA only)	$t+3$	Acknowledge the operation’s status.	0

Appendix

Table 7: Illustrative exact-step targets. The label applies to the indexed action, rather than to all actions up to that horizon.

Condition	Policy source	Question addressed
Full	Rules associated with the current task.	Can the WM detect violations when the relevant requirements are explicit?
Train-derived	Constraints extracted from a training subset.	Does extracted policy information support prediction beyond its extraction instances?
Generic	Non-instance-specific embodied-safety guidance.	How much can prediction rely on general safety guidance?

Appendix

Table 8: Policy inputs and the comparisons they enable.

Benchmark	ID	Prediction mode	Policy input	Horizons
HA	HA-1	Direct	Full	$h_{1}$ – $h_{6}$
	HA-2	Direct	Train-derived	$h_{1}$ – $h_{6}$
	HA-3	Iterative	Full	$h_{1}$ – $h_{6}$
	HA-4	Iterative	Train-derived	$h_{1}$ – $h_{6}$
IS-Bench	IS-1	Direct	Full	$h_{1}$ – $h_{3}$
	IS-2	Direct	Train-derived	$h_{1}$ – $h_{3}$

Appendix

Table 9: Complete prediction setting grid. HA has

2\times 2=4

conditions and IS-Bench has

2\times 3=6

. IDs are descriptive shorthand; every condition is evaluated at all listed horizons.

Figure 5: DeepSeek-V4-Flash on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 6: Qwen-AgentWorld on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 7: Qwen3.5-Plus on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 8: Qwen3.5-27B on HA: Direct and Iterative with full and train-derived policy inputs over

h_{1}

–

h_{6}

Figure 9: Qwen3.5-27B on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 10: GLM-5.3-Flash on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 11: GPT-5.6-Terra on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Figure 12: Gemini-3.7-Flash on IS-Bench: Direct and Iterative with full, train-derived, and generic policy inputs over

h_{1}

–

h_{3}

Condition	Explanatory text	Risk judgment	Risk score	Policy/action guidance
Normal	Original	Original	Original	Original
Empty	Removed	Retained	Removed	Removed
Irrelevant	Unrelated text	Retained	Removed	Removed
Corrupt	Key text damaged	Retained	Retained	Retained
Filler	Non-diagnostic filler	Retained	Removed	Removed
Full Inversion	Risk direction reversed	Reversed	Across threshold	Transformed with the risk assessment

Appendix

Table 10: Feedback fields retained, removed, or transformed under each intervention.

Case	Task	Mode	Recorded behavior
1	dl-t11	NP	Compares waiting with immediate authorization.
2	si-t4	NP	Anticipates identity exposure and publication prerequisites.
3	Cleaning fan	NP	Turns off the fan before cleaning.
4	si-t27	PI	Treats a zero-risk forecast as publication confirmation.
5	fin-t26	PI	Repeats a prepared filing after zero-risk advice.
6	si-t12	PI	Retains the proposed dispute after a zero-risk forecast.

Appendix

Table 11: Index of the seven qualitative cases. NP, PI, and PC denote No Prediction, Prediction-Informed, and Prediction-Constrained, respectively.

Figure 13: Feedback interventions for GLM-5.3-Flash on HA. Bars report SAR, AVS, and TCR on the 0–100 scale (higher is better), with labels rounded to one decimal place. Normal uses unperturbed Prediction-Constrained feedback; the intervention definitions are given in Appendix D .

Figure 14: Feedback interventions for Gemini-3.7-Flash on HA. Bars report SAR, AVS, and TCR on the 0–100 scale (higher is better), with labels rounded to one decimal place. Normal uses unperturbed Prediction-Constrained feedback; the intervention definitions are given in Appendix D .

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 341 papers

Aug 6, 2026

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

As large language model (LLM) agents increasingly invoke external tools and interact with real-world systems, unsafe actions may cause irreversible consequences on external states, user data, and downstream services. Recent runtime guardrails mitigate such risks by checking proposed actions before execution, but many remain reactive: they primarily assess the apparent safety of the current action, lacking an explicit model of how risk evolves across the trajectory. This limitation creates a critical blind spot for long-horizon risks, where individually benign-looking actions can gradually drift the agent toward hazardous states. In response, we propose DreamGuard, a proactive guardrail for LLM agents built around a risk-aware world model. The world model maintains a compact recurrent latent state over the trajectory and predicts future latent states from which DreamGuard derives immediate-hazard and prefix-risk evidence. It then fuses these multi-horizon signals into intervention decisions before execution. Experiments across four benchmarks and an online guardrail evaluation show that DreamGuard outperforms generic, reactive, and proactive guardrail baselines, achieves the best safety-utility trade-off among evaluated guardrails, and maintains an average end-to-end latency of 25 ms per call.

Wenhao Lin, Chenyu Yu, Xingwei Lin +6

Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University

LLM World Models LLM Guardrails World Models Runtime Enforcement for AI Agents LLM Agent Safety Long-Horizon LLM Agents

Ask PDF

Jun 3, 2026

PreAct-Bench: Benchmarking Predictive Monitoring in LLMs

Large language models (LLMs) are increasingly deployed as autonomous agents capable of executing multi-step action trajectories toward a given objective. While existing safety research has focused on detecting unethical behavior from complete trajectories, this paradigm is fundamentally retrospective: it identifies harm only after it has already occurred. In this work, we study a critical yet overlooked safety task, which we term Predictive Monitoring: given only a partial action trajectory, can a model infer whether it will culminate in an unethical action before the overt action is executed? To support this task, we present PreActBench, a benchmark of 1,000 paired ethical and unethical action trajectories spanning five domains. We evaluate a range of LLMs, safety guardrail models, and latent probing methods across varying fractions of the action trajectory using our Prefix Foresight F1 metric. Results show that while humans achieve promising performance, predictive monitoring remains challenging even for strong models, highlighting the need for future-oriented risk reasoning in LLM safety.

Hainiu Xu, Italo Luis da Silva, Jiangnan Ye +7

King’s College London · National University of Singapore · Southern University of Science and Technology +3

LLM Safety Benchmarks Moral Reasoning in Language Models AI Agent Safety AI Agent Benchmarks LLM Safety Evaluation AI Agent Safety Benchmarks

Ask PDF

Oct 8, 2026

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identifying forbidden actions alone is insufficient to ensure agent safety. In this paper, we argue that agent safety also depends on identifying required yet unperformed safety-critical actions, which we call obligations. Our preliminary study on a popular benchmark for evaluating safety shows that 56.92% of GLM-5.3 trajectories contain unfulfilled obligations, compared with only 30.00% containing forbidden actions. This finding reveals unfulfilled obligations as a major and previously overlooked source of safety risk. However, to our knowledge, no existing benchmark evaluates whether guard models can identify these obligations. To close this gap, we introduce ObligationBench, the first benchmark for evaluating the capability of obligation identification, comprising 240 expert-validated trajectories covering issue resolution, feature development, and terminal operations. Our evaluation of 14 representative models reveals substantial limitations: the highest recall and exact-match rate are only 48.97% and 10.00%, respectively. To address these limitations, we develop ObligationGuard using 40,000 synthetic training examples. ObligationGuard achieves 57.52% recall and an exact-match rate of 21.67%, surpassing all evaluated models on both metrics. We call on the community to incorporate obligation identification into the design and evaluation of future guard models to improve agent safety.

Youwei Feng, Yitong Zhang, Yuetong Liu +1

Tsinghua University, Beijing, China · College of AI, Tsinghua University, Beijing, China

LLM Agent Safety LLM Guardrails AI Agent Safety Benchmarks LLM Agent Evaluation

Ask PDF

Abstract

LLM Agent Safety

Explore similar work

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

PreAct-Bench: Benchmarking Predictive Monitoring in LLMs

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models