Beyond Ensemble Disagreement: Predictive STL Monitoring of RL Locomotion under Distribution Shift

Qurban Ali; Badaruddin Chachar; Maqsood Ali; Sajjad Ali Rajper

cs.RONov 1, 2026

Beyond Ensemble Disagreement: Predictive STL Monitoring of RL Locomotion under Distribution Shift

Authors: Qurban Ali, Badaruddin Chachar, Maqsood Ali, Sajjad Ali Rajper

Organizations: Sindh Madressatul Islam University, Karachi, Pakistan · Shah Abdul Latif University, Ghotki, Pakistan · University of Camerino, Camerino, Italy · Institute of Biophysics, CNR, Palermo, Italy ·

Abstract

RL locomotion policies trained under nominal simulation conditions may encounter distribution shifts that compromise safety. Predictive monitors could anticipate violations, but their world-model forecasts may become unreliable under shift, and ensemble disagreement is not calibrated. We evaluate predictive Signal Temporal Logic (STL) monitoring for frozen Ant-v5 and HalfCheetah-v5 policies using closed-loop world-model rollouts and a three-model ensemble. Across external-force, observation-noise, and mass shifts, predicted robustness ranks HalfCheetah's future STL violations strongly, whereas disagreement does not reliably track observation prediction error or violations. Under severe noise, violations remain common ( $64.2\pm24.4\%$ at $K{=}20$ across three PPO seeds), yet current-state robustness without prediction comes within $0.03$ AUROC of predictive robustness. Penalizing predicted robustness by disagreement does not consistently improve ranking. Ant provides complementary per-specification evidence, but its composite target has no positive violations and its any-violation target is single-class in most evaluated seed-condition slices. Overall, predictive robustness provides useful STL-specific ranking but little demonstrated additional ranking value over the current state; these findings caution against treating ensemble disagreement as a reliable indicator of safety uncertainty.

Figures & tables

Panel of Figure 1 — Figure 1: Predictive STL monitoring pipeline. The current observation $o_{t}$ feeds the frozen PPO policy and the world-model ensemble, whose members each generate a $K$ -step closed-loop rollout by repeatedly querying the policy on predicted observations. Predicted trajectories are evaluated with STL and aggregated into predictive robustness $P$ and STL disagreement $U_{\rho}$ ; independently, $O$ is evaluated on the observed trajectory $\tau_{t:t+K}$ , and $S_{\mathrm{UP}}$ penalizes $P$ by $U_{\rho}$ .

		AUROC		AUPRC
Condition	Viol%	$U_{\rho}$	$-P$	$U_{\rho}$	$-P$
baseline	$56{\pm}25$	$0.50{\pm}0.08$	$\mathbf{0.92{\pm}0.08}$	$0.59{\pm}0.26$	$\mathbf{0.97{\pm}0.02}$
noise 0.05	$57{\pm}29$	$0.50{\pm}0.07$	$\mathbf{0.81{\pm}0.25}$	$0.64{\pm}0.24$	$\mathbf{0.95{\pm}0.05}$
noise 0.15	$56{\pm}29$	$0.56{\pm}0.18$	$\mathbf{0.98{\pm}0.00}$	$0.61{\pm}0.28$	$\mathbf{0.97{\pm}0.03}$
mass 0.8x	$55{\pm}24$	$0.47{\pm}0.09$	$\mathbf{0.96{\pm}0.05}$	$0.63{\pm}0.19$	$\mathbf{0.98{\pm}0.02}$
mass 1.6x	$63{\pm}28$	$0.47{\pm}0.03$	$\mathbf{0.94{\pm}0.01}$	$0.47{\pm}0.04$	$\mathbf{0.92{\pm}0.03}$

Table 1: Failure prediction at

K{=}10

for the HalfCheetah any-violation target (mean

\pm

std of per-seed metrics over 3 PPO seeds, 10 episodes per seed). Viol% is multi-seed any-violation prevalence, not comparable to seed-0 per-spec rates in Appendix C . Ant-v5 is not reported in this multi-seed table because its any-violation target is single-class (

79

–

100\%

positive) in 21 of 24 evaluated seed-condition slices (8 conditions

\times

3 seeds); the three slices admitting both classes give AUROC(

-P

)

0.98

–

1.00

(Tab. 9 ). Its composite target has no positive violations at all. Predicted robustness (

-P

) substantially outperforms the evaluated disagreement measures.

Panel of Figure 2 — Figure 2: HalfCheetah any-violation failure prediction (descriptive mean $\pm$ std over 3 seeds). Left ( $K{=}10$ , Tab. 1 ): $U_{\rho}$ stays near chance (0.47–0.56) while $-P$ ranks failures strongly (0.92–0.98); $U_{o}$ / $U_{z}$ are likewise near chance (0.44–0.54). Right ( $K{=}20$ , severe noise, 64.2 $\pm$ 24.4% prevalence): $U_{\rho}$ / $U_{o}$ / $U_{z}$ give $0.65$ / $0.47$ / $0.46$ versus $\mathbf{0.99}$ for $-P$ . Dashed line marks chance.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Ant-v5
	Composite STL
Condition	$O$	$P$	$\delta_{\rho}$	$U_{\rho}$	$r_{UE}$ ( $n$ )	$r_{UV}$	Viol%
baseline	$+0.75$	$+0.71$	$0.04$	$0.049$	$-0.75$ (3)	—	0.0
force 5N	$+0.75$	$+0.73$	$0.02$	$0.056$	$+0.49$ (3)	—	0.0
force 15N	$+0.74$	$+0.72$	$0.07$	$0.043$	$-0.36$ (19)	—	0.0
noise 0.05	$+0.66$	$+0.70$	$0.05$	$0.035$	$-0.95$ (5)	—	0.0

Appendix

Table 2: Legacy single-seed predictive monitoring summary at

K{=}10

(closed-loop, environment-dependent property composite).

O

: retrospective reference score evaluated on the actually observed future trajectory.

P

: predictive score evaluated on the world-model trajectory.

\delta_{\rho}

: mean

|\Delta_{\rho}|

U_{\rho}

: mean STL robustness disagreement.

r_{UE}

: Pearson correlation between STL robustness disagreement and prediction error (

n

denotes monitoring windows; small

n

values result from early episode termination in shifted conditions). Viol%: future violation rate.

Environment	$K{=}1$	$K{=}5$	$K{=}10$	$K{=}20$
Ant-v5	0.30	0.31	0.32	0.34 ∗
HalfCheetah-v5	2.52	2.49	2.48	2.50

Appendix

Table 3: Mean observation prediction error (

L_{2}

) by prediction horizon, closed-loop rollouts, averaged over all conditions. ∗ Ant-v5

K{=}20

is reported only for the subset of conditions with sufficient valid monitoring windows.

Environment	Metric	$K{=}1$	$K{=}5$	$K{=}10$	$K{=}20$
	Horizon $K$
Ant-v5	$O$	$+0.82$	$+0.77$	$+0.75$	—
Ant-v5	$P$	$+0.82$	$+0.78$	$+0.71$	—
HalfCheetah-v5	$O$	$+0.17$	$+0.13$	$+0.11$	$+0.10$
HalfCheetah-v5	$P$	$+0.18$	$+0.16$	$+0.14$	$+0.16$

Appendix

Table 4: Retrospective (

O

) versus predictive (

P

) composite STL score at each horizon, nominal condition, closed-loop. Ant-v5 uses five predictive properties; HalfCheetah-v5 uses three (pitch, height, contact; recovery excluded at

K{<}50

Panel of Figure 3 — Figure 3: Predictive bias $(P-O)$ versus prediction horizon for representative conditions. Ant-v5 exhibits mixed bias directions: predictive robustness is mildly pessimistic under the 15N force shift but increasingly optimistic under severe observation noise. HalfCheetah-v5 is consistently optimistic, with the predictive–retrospective gap increasing with horizon and becoming largest under observation noise.

Ant-v5
	Horizon $K$
Condition	$K{=}1$	$K{=}5$	$K{=}10$	$K{=}20$
baseline	$+0.10$	$-0.58$	$-0.75$	—
force 5N	$+0.13$	$-0.85$	$+0.49$	—
force 15N	$-0.12$	$-0.29$	$-0.36$	$-0.39$
noise 0.05	$+0.12$	$-0.05$	$-0.95$	—

Appendix

Table 5: Pearson correlation (

r

) between STL robustness disagreement and observation prediction error, closed-loop. All values are descriptive; correlations with small sample sizes (

n<10

) should be interpreted with particular caution.

Ant-v5: violation rate (%)
	Horizon $K$
Condition	$K{=}1$	$K{=}5$	$K{=}10$	$K{=}20$
All conditions	0.0	0.0	0.0	0.0
HalfCheetah-v5: correlation ( $r$ )
baseline	$-0.07$	$-0.04$	$-0.20$	$-0.37$
force 5N	$-0.01$	$-0.14$	$-0.26$	$-0.57$

Appendix

Table 6: Correlation (

r

) between STL robustness disagreement and future STL violation, and legacy single-seed window-level composite violation rate (%) by horizon, closed-loop. The Ant-v5 violation rates refer to the predictive composite and are zero at all reported horizons. The legacy single-seed runs used 3 episodes per condition (

n{=}27

windows at

K{=}10

), whereas the seed-0 horizon analysis in Appendix C (Tab. 10 ) uses 10 episodes (

n{=}90

); the composite-label rates for noise

\sigma{=}0.15

K{=}10

(

37.0\%

vs.

35.6\%

) therefore differ by sampling, not by definition.

Panel of Figure 4 — Figure 4: Left: Future violation rate for HalfCheetah-v5 at $K=20$ across representative distribution shifts. Severe observation noise produces the highest violation rate (66.7%). Right: Point-biserial correlation $r_{UV}$ between ensemble STL robustness disagreement and future violation across the same conditions. The correlations vary in both magnitude and direction; notably, the condition with the highest violation rate has only a moderate positive correlation ( $r_{UV}=0.27$ ), while several lower-violation conditions exhibit negative correlations. These condition-level results provide no consistent evidence that ensemble STL robustness disagreement reliably identifies future STL violations.

Ant-v5 (force 15N)
	$\bar{\rho}$	$\lambda U_{\rho}$	$S_{\mathrm{UP}}$
$\lambda{=}0$	$+0.72$	$0.000$	$+0.72$
$\lambda{=}1$	$+0.72$	$0.045$	$+0.67$
$\lambda{=}2$	$+0.72$	$0.090$	$+0.63$
$\lambda{=}5$	$+0.72$	$0.226$	$+0.49$
HalfCheetah-v5 (noise $\sigma{=}0.15$ )

Appendix

Table 7:

S_{\mathrm{UP}}

components,

K{=}20

, closed-loop. Ant-v5 under force 15N; HalfCheetah-v5 under noise

\sigma{=}0.15

	Ant-v5		HalfCheetah-v5
Condition	no_falls	recovery	no_falls	recovery
baseline	88.9	88.9	90.0	64.4
noise 0.15	100	100	95.6	50.0
mass 1.6x	79.2	75.0	98.9	76.7
friction 0.5x	100	100	95.6	66.7
compound	100	100	98.9	75.6

Appendix

Table 8: Per-specification (binary, each spec alone) violation rate (%) at

K{=}10

, seed 0 (10 episodes). Recovery robustness is computed on the truncated

K

-step window; the full 50-step interpretation is available only at

K{\geq}50

. These per-spec rates use their own single-spec targets and are not comparable to the composite-label rate in Tab. 10 or the any-violation rate in Tab. 1 (same seed-0 windows under noise 0.15: height 95.6%, composite 35.6%, any-violation 95.6%). Composite masks per-spec failures: Ant composite 0% while no_falls /recovery violate frequently; HalfCheetah violations concentrate in no_falls /recovery while stability_pitch / contact_safety are near-constant.

Condition	seed	$n$	Viol%	$-P$	$U_{\rho}$	$-O_{t}$
baseline	0	36	88.9	1.00	0.76	0.81
mass 1.6x	0	24	79.2	0.98	0.83	0.87
friction 0.5x	42	44	95.5	1.00	0.42	1.00

Appendix

Table 9: Ant-v5 any-violation target: the three seed-condition slices (

K{=}10

) whose labels admit both classes; all other evaluated slices are

100\%

positive (single class). AUROC is undefined for the remaining slices. The any-violation label is the union of all evaluated per-specification labels, including truncated recovery labels. Seeds are PPO training seeds

\{0,42,123\}

Panel of Figure 5 — Figure 5: Horizon sensitivity under severe noise (HalfCheetah noise $\sigma{=}0.15$ , seed 0; values from Tab. 10 ). Longer horizons expose more violations (10.2% at $K{=}1$ to 67.5% at $K{=}20$ ), yet $U_{\rho}$ AUROC climbs only from 0.15 toward 0.41 while $-P$ degrades gracefully from 0.90 to 0.71. Dashed gray line marks chance (0.5). Single-seed illustration of the pattern reported multi-seed at $K{=}10$ / $20$ in Sec. 7.1 (Tab. 6 reports 66.7% at $K{=}20$ on the legacy single-seed windowing).

$K$	$n$	Viol%	AUROC( $U_{\rho}$ )	AUROC( $-P$ )	AUPRC( $U_{\rho}$ )	AUPRC( $-P$ )
1	990	10.2	0.15	0.90	0.06	0.74
5	190	23.7	0.29	0.79	0.17	0.67
10	90	35.6	0.36	0.73	0.28	0.63
20	40	67.5	0.41	0.71	0.64	0.82

Appendix

Table 10: Horizon sensitivity under severe noise (HalfCheetah noise

\sigma{=}0.15

, seed 0). Viol% is the composite-label rate (fraction of windows with negative observed composite), not the per-specification or any-violation rates of Tab. 8 /Tab. 1 ; on the same windows the any-violation rate is higher (e.g. 95.6% at

K{=}10

-P

ranks failures across horizons while

U_{\rho}

remains at or below chance. Baseline shows the same pattern at lower prevalence.

	All windows (AUROC)			Safe-at-start
Condition	$-O_{t}$	$-P$	$U_{\rho}$	Safe%	viol%	$-O_{t}$	$-P$	$U_{\rho}$
baseline	$0.90{\pm}0.09$	$0.92{\pm}0.08$	$0.50{\pm}0.08$	$63{\pm}12$	$36{\pm}31$	$0.90{\pm}0.09$	$0.90{\pm}0.10$	$0.38{\pm}0.11$
noise 0.05	$0.79{\pm}0.26$	$0.81{\pm}0.25$	$0.50{\pm}0.07$	$63{\pm}11$	$38{\pm}38$	$0.76{\pm}0.29$	$0.76{\pm}0.30$	$0.38{\pm}0.21$
noise 0.15	$0.97{\pm}0.01$	$0.98{\pm}0.00$	$0.56{\pm}0.18$	$65{\pm}7$	$36{\pm}40$	$0.93{\pm}0.04$	$0.95{\pm}0.05$	$0.52{\pm}0.24$
mass 0.8x	$0.93{\pm}0.07$	$0.96{\pm}0.05$	$0.47{\pm}0.09$	$63{\pm}10$	$32{\pm}28$	$0.92{\pm}0.11$	$0.91{\pm}0.12$	$0.36{\pm}0.03$
mass 1.6x	$0.92{\pm}0.02$	$0.94{\pm}0.01$	$0.47{\pm}0.03$	$60{\pm}10$	$45{\pm}40$	$0.86{\pm}0.06$	$0.87{\pm}0.06$	$0.62{\pm}0.12$

Appendix

Table 11: Current-state baseline and safe-at-start analysis, HalfCheetah-v5,

K{=}10

, mean

\pm

std over 3 seeds (any-violation target).

-O_{t}

: instantaneous robustness at time

t

(no prediction). Safe: all pointwise predicates satisfied at

t

; “safe viol.%” is the future (

t{+}1

t{+}K

) any-violation rate within that subset. SAFE columns: AUROC restricted to safe-at-start windows with the future-only label.

	baseline			noise 0.15
$h_{\min}$	viol%	$-P$	$U_{\rho}$	viol%	$-P$	$U_{\rho}$
0.08	$54.8{\pm}23.7$	$0.90{\pm}0.10$	$0.51{\pm}0.09$	$55.9{\pm}28.6$	$0.98{\pm}0.00$	$0.56{\pm}0.18$
0.10	$56.3{\pm}25.0$	$0.92{\pm}0.08$	$0.50{\pm}0.08$	$55.9{\pm}28.6$	$0.98{\pm}0.00$	$0.56{\pm}0.18$
0.12	$57.0{\pm}26.0$	$0.94{\pm}0.05$	$0.48{\pm}0.05$	$56.3{\pm}28.3$	$0.99{\pm}0.00$	$0.56{\pm}0.18$

Appendix

Table 12: Height-threshold sensitivity ( no_falls

h_{\min}

), HalfCheetah-v5,

K{=}10

, mean

\pm

std over 3 seeds, any-violation target.

$K{=}10$
$\lambda$	AUROC( $-S_{\mathrm{UP}}$ )	Flag rate %
0	$0.983{\pm}0.003$	$5.2{\pm}1.0$
1	$0.983{\pm}0.002$	$5.2{\pm}1.0$
2	$0.986{\pm}0.004$	$5.9{\pm}1.0$
5	$0.989{\pm}0.004$	$8.1{\pm}3.8$
$K{=}20$

Appendix

Table 13: Disagreement-penalized score

S_{\mathrm{UP}}{=}P{-}\lambda U_{\rho}

and risk–coverage on the HalfCheetah any-violation target (noise

\sigma{=}0.15

, mean

\pm

std over 3 seeds). Flag rate is the fraction of windows with

S_{\mathrm{UP}}<0

(the fixed STL satisfaction boundary, applied uniformly across conditions and horizons). Penalization is flat (

\Delta

AUROC

{\leq}0.01

) while flag rate rises; abstention shows little change across coverage levels.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 232 papers

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer
Nov 2, 2026Weixiang Zhao, Zhihao Liu, Guancheng Li +3LLM Agent Safety Safety Filtering
Harbin Institute of Technology
Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields
Oct 14, 2026Siwei Ju, Jan Peters, Oleg ArenzSafe RL Safety Filtering
Department of Computer Science, Technical University of Darmstadt, Germany. · Hessian.AI. · German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning. +1
Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering
Jun 5, 2026Aditya Shirwatkar, Sebastian Sanokowski, Shishir Kolathaya +2Robot Safety Safety Filtering
Robert Bosch Center for Cyber Physical Systems, Indian Institute of Science, Bangalore, India · Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, Munich, Germany · Department of Computer Science & Automation, Indian Institute of Science, Bangalore, India +2

Nov 2, 2026J/K move · Enter open · S save

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao+2

Harbin Institute of Technology

LLM Agent Safety Safety Filtering Multi-Turn Language Model Safety Evaluation World Models

Open paper AskPDF arXiv

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.

Abstract

Safe RL

Explore similar work

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields

Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering