Vision--Language Signals in Constrained RL: Safety Gains Without Anticipation

Samuel Tetteh; Cody Fleming

cs.LGSep 28, 2026

Vision--Language Signals in Constrained RL: Safety Gains Without Anticipation

Authors: Samuel Tetteh, Cody Fleming

Organizations: Iowa State University Ames, Iowa, USA

Abstract

Safe reinforcement learning seeks policies that maximise task performance while satisfying safety constraints. In driving benchmarks, however, collision costs typically appear only at the time of collision, providing no advance warning of an approaching hazard. Frozen vision--language models can provide dense semantic feedback, yet it remains unclear whether their scores anticipate collisions and which component drives an observed safety improvement. Episodic cost can also favour policies that make little task progress. To address these gaps, we propose VLM-Safe-RL, a framework that integrates frozen CLIP signals into PPO-Lagrangian through reward shaping and an augmented multiplier update. On MetaDrive Hard, which combines the densest traffic with the largest map, the catastrophe rate falls from 31.6% to 19.4%. FormulaOne-L2 analysis finds no evidence that the CLIP signals anticipate collisions and shows that the VLM term has a negligible effect on the Lagrange multiplier. These findings show a conditional reduction in observed catastrophe rate without evidence of collision anticipation.

Figures & tables

Figure 1: VLM-Safe-RL and its measured components. (1) The policy acts from vector observations, and frozen CLIP scores each rendered frame against positive and negative prompts on separate paths, every fourth step on MetaDrive and Bullet. (2) The gated positive score, in green, adds a bonus to the reward. The epoch mean negative score, in red and dashed, enters the multiplier update through $\eta_{2}$ , the route that could carry advance warning. (3) PPO-Lagrangian omits both terms. Yellow notes give the measured effect of each component.

Environment	Cat. %		Viol. %		Cat. $\Delta$ 95% CI
	PPOLag	VLM $+$ Conf	PPOLag	VLM $+$ Conf	(pp)
MetaDrive Easy	14.0	32.7	18.0	38.7	$[+5.3,+35.3]$
MetaDrive Medium	26.0	28.0	32.8	36.0	$[-8.8,+14.0]$
MetaDrive Hard	31.6	19.4	39.2	26.0	$[-21.8,-3.2]$
Bullet Car-Reach	13.2	12.7	20.8	19.8	$[-3.1,+1.9]$
F1-L2 †	2.5	8.0	17.5	18.0	n/a

Table 1: Cross-environment evaluation. MetaDrive uses

50

deterministic episodes per run on seeds

10000

–

10049

, which map into the training scenario pool. Bullet uses

200

deterministic-policy episodes per run, but its reset states cannot be seeded. Each method has

3

5

and

10

MetaDrive runs on Easy, Medium and Hard, and

6

Bullet runs. Rates are pooled over episodes. Catastrophe denotes

C(\tau){>}4d

, and violation denotes

C(\tau){>}d

. The

95\%

interval is a

10{,}000

-resample seed-level bootstrap of (VLM

+

Conf)

-

PPOLag in percentage points. The F1-L2 row uses the first

20

episodes from five calibrated VLM

+

Conf runs and two matched PPOLag runs. We do not interpret that row as safety (§ 7 ).

Figure 2: Lagrange multiplier on FormulaOne-L2. The figure compares VLMPPOLag with

\eta_{2}{=}0.01

(red), its matched

\eta_{2}{=}0

ablation (purple, dashed), and its matched confidence-gated variant (green). All arms use seeds

\{42,123,456\}

and score every frame. Bold lines show seed means, bands show

\pm 1

population standard deviation, and thin lines show individual seeds. The purple mean ends at epoch

39

, the last epoch reached by its seed-

42

run.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Hyperparameter	Value
Optimisation	Total timesteps	$10^{6}$ (FormulaOne, MetaDrive, Bullet), $2{\times}10^{6}$ (Bullet long-horizon runs)
	Steps per epoch	$20{,}000$ ( $50$ epochs for $10^{6}$ -step runs, $100$ epochs for the Bullet $2{\times}10^{6}$ -step runs)
	Discount $\gamma$	$0.99$
	GAE $\lambda_{\text{GAE}}$	$0.95$
	Learning rate (actor / critic)	$3{\times}10^{-4}$ / $3{\times}10^{-4}$ , CPO-family runs have no actor learning rate and a $10^{-3}$ critic learning rate
	Linear learning-rate decay	on

Appendix

Table 3: Hyperparameters for the reported FormulaOne, Bullet and MetaDrive runs. Values that differ across methods or run sets are listed explicitly.

Env.	Method	$k_{\text{clip}}$	Runs	Time (h)	Rollout (h)	Steps/s	vs. ref.
FormulaOne-L2	PPOLag, 10 iter.	none	3	18.6 [18.4, 18.7]	18.4	14.9	$-3\%$
	PPOLag, 40 iter.	none	2	19.2 [19.2, 19.2]	18.2	14.5	ref.
	CPPOPID	none	3	19.2 [19.1, 19.2]	18.2	14.5	$0\%$
	PPOLag-Dec.	1	2	23.1 [22.9, 23.2]	22.2	12.0	$+20\%$
	VLMPPOLag	1	3	23.4 [23.0, 24.0]	22.5	11.9	$+22\%$
	$+$ Conf, prior	1	3	23.3 [23.1, 23.5]	22.3	11.9	$+21\%$

Appendix

Table 4: Training time per completed run. We report the mean and range of total wall-clock time, mean rollout time accumulated across epochs, and mean environment steps per second. The final column compares each method with the VLM-free PPOLag reference in the same environment. FormulaOne-L2 includes PPOLag configurations with

10

and

40

update iterations. PPOLag-Dec. denotes PPOLag-Decoupled, excluding its incomplete seed-

42

run. The two

+

Conf rows use the prior-symmetric and calibrated gates. A CLIP period of “none” denotes runs without frame scoring.

Figure 3: MetaDrive catastrophe rates for deterministic PPOLag and VLMPPOLag $+$ Conf policies. The earlier evaluation uses $20$ episodes per run, a shared simulator, randomised traffic and a scenario pool of $100$ . The final protocol uses $50$ episodes per run, fixed traffic, a new simulator for each episode and a pool of $10{,}000$ . Both settings evaluate scenarios from their training pool. Bars show mean per-run rates with population standard deviations. Easy and Medium use $3$ and $5$ runs per method in both settings. Hard uses $3$ runs in the earlier setting and $10$ in the final protocol.

Positive prompts ( $K{=}4$ )	Negative prompts ( $L{=}4$ )
FormulaOne
“the racecar is centered on the track and driving safely”	“the racecar is about to crash into the barrier”
“the racecar is following the track perfectly with smooth steering”	“the racecar is off the track and unsafe”
“the racecar is making progress along the racing line”	“the car is colliding with obstacles”
“the car is driving efficiently without collisions”	“the car is driving in the wrong direction”
MetaDrive

Appendix

Table 5: Default prompt sets used for CLIP scoring.

Figure 4: Catastrophe rates for PPOLag and VLMPPOLag $+$ Conf under the v1, v2 and v3 prompt sets. Bullet uses three training runs per method and $200$ evaluation episodes per run. MetaDrive uses $50$ in-distribution episodes per run with three training runs on Easy and ten on Hard. Bars show mean per-run rates, error bars show population standard deviations, and dashed lines mark the PPOLag mean.

	SafeVLA ( Zhang et al., 2025 )	VLM-RL ( Huang et al., 2024 )	CPO-CLG † ( Huang et al., 2024 )	Ours
Safety formulation	CMDP	none	CMDP	CMDP
CLIP scoring	N/A	CLG difference	CLG difference	separate cosines
VLM role	policy	reward	reward	reward and dual mean
Lagrange multiplier	yes	no	no	yes
VLM term in multiplier	no	no	no	yes (negligible)
Confidence gating	no	no	no	yes

Appendix

Table 6: Comparison of VLM-augmented approaches. Ours denotes VLMPPOLag

+

Conf. † denotes the CPO adaptation of the VLM-RL CLG reward, used as a baseline.

	Logged $J_{R}$	Bonus	Environment return
Arm	final	final	last-5	last-10	final
CPO-Coupled	21.62	21.28	0.381	0.367	0.337
CPO-Decoupled	63.88	63.63	0.398	0.318	0.242
PPOLag-Decoupled	63.77	63.54	0.318	0.355	0.228
VLMPPOLag	63.83	63.50	0.319	0.397	0.328
PPOLag (unmatched)	0.74	0	0.537	0.509	0.744

Appendix

Table 8: Decomposition of the logged training return on FormulaOne-L2 (seeds

\{42,123,456\}

). Logged

J_{R}

and bonus

\lambda_{r}\,\overline{r_{\text{vlm}}}\,T

are final-epoch seed means. Environment return is the logged return minus the bonus, averaged over the last

5

10

logged epochs or taken at the final epoch, then averaged over seeds. Runs that stopped early use their last logged epochs (CPO-Coupled seeds

42

and

456

, CPO-Decoupled seed

42

, PPOLag-Decoupled seed

42

). The VLM-free PPOLag (unmatched) logs its environment return directly.

Figure 5: Training curves on FormulaOne-L0/L1/L2 for seven methods (PPO, CPO, PPOLag, CPO-CLG, CPO-Decoupled, VLMPPOLag and VLMPPOLag

+

Conf),

3

seeds

\{42,123,456\}

each: logged episode return

J_{R}

(top row) and environment cost

J_{C}

(bottom row) per epoch. For the VLM-shaped arms (CPO-CLG, CPO-Decoupled, VLMPPOLag and VLMPPOLag

+

Conf)

J_{R}

is the augmented return, which includes the CLIP shaping bonus, so the top row does not compare task performance across arms. Bold lines are seed means, shaded bands

\pm 1

population standard deviation over seeds, and faded lines individual seeds. The grey dashed line marks

d{=}25

. VLMPPOLag

+

Conf here is the three-seed prior-symmetric set at

k_{\text{clip}}{=}1

, not the calibrated five-seed cells of Table 7 . PPOLag uses the unmatched VLM-free configuration. Curves are truncated to the shortest run in each cell, so CPO-Decoupled on L2 ends at epoch

25

Group 1	Group 2	$(n_{1},n_{2})$	$H_{1}$	$\bar{x}_{1}$	$\bar{x}_{2}$	$\Delta$	$p$
Environment cost $J_{C}$
VLMPPOLag $+$ Conf ∗	VLMPPOLag	$(5,3)$	$<$	$22.54$	$43.70$	$-21.16$	$\mathbf{1/56}$
VLMPPOLag $+$ Conf ∗	PPOLag-Decoupled	$(5,3)$	$<$	$22.54$	$43.75$	$-21.21$	$\mathbf{1/56}$
VLMPPOLag $+$ Conf ∗	PPOLag-RND	$(5,3)$	$<$	$22.54$	$46.73$	$-24.19$	$\mathbf{1/56}$
VLMPPOLag	PPOLag-Decoupled	$(3,3)$	$<$	$43.70$	$43.75$	$-0.05$	$10/20$
Logged return $J_{R}$ (augmented for VLM-shaped arms)

Appendix

Table 9: Exact one-sided permutation tests on FormulaOne-L2. Each run contributes its mean over the final

10

logged epochs. The calibrated group contains five seeds, while each comparator contains three.

\bar{x}_{1}

and

\bar{x}_{2}

are the group means and

\Delta=\bar{x}_{1}-\bar{x}_{2}

. The minimum attainable

p

-value is

1/56

for comparisons with group sizes

(5,3)

(3,5)

and

1/20

for

(3,3)

. Bold values attain this minimum. Environment cost tracks distance travelled on this task, and logged return includes the shaping bonus for VLM-shaped methods. a PPOLag-RND instead reports environment return.

Method	$d$	$J_{R}$ (augmented)	$J_{C}$ final	$J_{C}$ last-10	Over budget
PPOLag-Decoupled	15	63.8	33.8	42.5	1/1
PPOLag-Decoupled	25	63.8	40.7	43.8	3/3
PPOLag-Decoupled	35	63.9	49.6	46.5	1/1
VLMPPOLag $+$ Conf ∗	25	31.8	22.4	22.5	1/5

Appendix

Table 10: PPOLag-Decoupled budget sweep on FormulaOne-L2.

J_{C}

is reported at the final epoch and as a mean over the final

10

epochs.

J_{R}

is the final-10-epoch augmented return. The

d\in\{15,35\}

entries are single seed-

42

runs with

k_{\mathrm{clip}}{=}4

. The

d{=}25

entry averages three runs with

k_{\mathrm{clip}}{=}1

, including one run that ended at epoch

39

. The calibrated five-seed method is included only as a reference. The rows are not matched.

Arm	Seed	Path	Progress	Cost	Return	Viol. %	Cat. %
PPO (no VLM)	42	5.278	2.265	346.30	1.522	74	68
	123	6.902	2.611	357.48	1.535	82	78
	456	4.889	2.006	322.30	1.292	66	58
CPO (no VLM)	42	2.977	0.673	41.44	0.097	18	12
	123	2.317	0.498	50.58	0.053	12	10
	456	1.393	0.430	24.68	0.177	12	8

Appendix

Table 11: Per-run FormulaOne-L2 task metrics from

50

deterministic episodes per run. Path length and net progress use simulator distance units. Return and cost are measured without VLM augmentation. Viol. denotes episodes with

J_{C}{>}d{=}25

, and Cat. denotes episodes with

J_{C}{>}4d

. No episode completes a route stage. ∗ The calibrated five-seed method. † An untrained initial policy, excluded from all arm-level summaries.

	Family A		Family B
Seed	Final	Last-10	Final	Last-10
42	17.75	16.14	21.37	21.57
123	36.37	22.39	28.16	21.58
456	21.70	32.63	13.61	12.13
789	23.07	24.11	27.00	21.45
1024	13.26	17.43	10.51	12.65

Appendix

Table 12: Training-time environment cost for two independent calibrated VLMPPOLag

+

Conf families on FormulaOne-L2. Final denotes epoch

50

, and Last-10 averages epochs

41

–

50

. Summary rows report the mean

\pm

population standard deviation over five seeds and the number satisfying

J_{C}{\leq}d{=}25

	L0		L1		L2
Method	$J_{R}$	$J_{C}$	$J_{R}$	$J_{C}$	$J_{R}$	$J_{C}$
FOCOPS	1.13 $\pm$ 0.39	0.0 $\pm$ 0.0	0.26 $\pm$ 0.08	45.1 $\pm$ 17.7	0.35 $\pm$ 0.33	27.6 $\pm$ 10.2
CUP	1.76 $\pm$ 0.36	0.0 $\pm$ 0.0	0.38 $\pm$ 0.17	46.9 $\pm$ 10.1	0.33 $\pm$ 0.09	33.0 $\pm$ 5.8
P3O	1.69 $\pm$ 0.59	0.0 $\pm$ 0.0	0.04 $\pm$ 0.35	133.0 $\pm$ 29.9	0.00 $\pm$ 0.18	58.7 $\pm$ 33.0

Appendix

Table 13: Additional constrained-RL baselines on FormulaOne L0–L2 without VLM augmentation. Each run contributes its mean over the final five of

50

epochs. Entries report the mean

\pm

population standard deviation over seeds

\{42,123,456\}

J_{R}

is environment return, and blue entries satisfy

J_{C}{\leq}d{=}25

Method	$n$	env $J_{R}$	env $J_{C}$	Viol. %	Cat. %	Stages	Path	Progress
PPO	3	1.450 $\pm$ 0.111	342.03 $\pm$ 14.68	74.0	68.0	0/7	5.69 $\pm$ 0.87	2.294 $\pm$ 0.248
CPO	3	0.109 $\pm$ 0.051	38.90 $\pm$ 10.73	14.0	10.0	0/7	2.23 $\pm$ 0.65	0.534 $\pm$ 0.102
PPOLag (unmatched)	3	0.626 $\pm$ 0.093	85.77 $\pm$ 15.72	22.7	19.3	0/7	3.47 $\pm$ 1.31	1.045 $\pm$ 0.196
PPOLag-Decoupled †	2	0.231 $\pm$ 0.040	21.64 $\pm$ 5.82	14.0	6.0	0/7	1.91 $\pm$ 0.10	0.566 $\pm$ 0.022
VLMPPOLag	3	0.240 $\pm$ 0.242	22.56 $\pm$ 7.56	16.0	7.3	0/7	2.14 $\pm$ 0.33	0.547 $\pm$ 0.171
VLMPPOLag $+$ Conf	5	0.125 $\pm$ 0.059	15.71 $\pm$ 5.52	13.2	4.4	0/7	2.31 $\pm$ 0.41	0.395 $\pm$ 0.082

Appendix

Table 14: Aggregate FormulaOne-L2 task metrics from

50

deterministic episodes per run without VLM augmentation.

n

denotes the number of trained policies. Values are means

\pm

population standard deviations over per-policy episode means. Violation and catastrophe rates are pooled over episodes using thresholds

J_{C}{>}d

and

J_{C}{>}4d

. Stages denotes completed route stages out of seven. Path length and net progress use simulator distance units. The matched PPOLag control was not evaluated under this protocol. † One untrained initial policy is excluded.

Configuration	$J_{R}$ (augmented)	$J_{C}$	Viol.
Prior-symmetric gate	48.4	30.5	1/3
Ungated, $\eta_{2}{=}0.01$	63.8	40.2	2/3
Ungated, $\eta_{2}{=}0$	63.8	40.7	3/3
Coupled CPO (unmatched)	21.6	32.4	3/3
VLM-free PPOLag (unmatched)	0.7	55.8	2/3

Appendix

Table 15: FormulaOne-L2 component comparisons over seeds

\{42,123,456\}

at the final logged epoch. Every VLM row scores each frame.

J_{R}

is augmented return and does not measure task performance. Viol. counts seeds with

J_{C}{>}d

. The first two rows differ only in confidence gating, and the second and third differ only in the VLM multiplier term. The final two rows change several settings and are included as unmatched references. Runs that ended early contribute their final available epoch.

	F1-L0 (final)		F1-L1 (final)		F1-L2 (last 10)
Seed	$J_{R}$	$J_{C}$	$J_{R}$	$J_{C}$	$J_{R}$	$J_{C}$
42	42.1	0.0	29.6	6.7	24.6	16.1
123	41.2	0.0	39.2	20.4	33.6	22.4
456	36.6	0.0	24.2	44.4	52.5	32.6
789	41.5	0.0	34.8	11.2	15.8	24.1
1024	61.2	0.0	39.9	21.4	32.6	17.4

Appendix

Table 16: Per-seed training metrics for calibrated VLMPPOLag

+

Conf on FormulaOne after

10^{6}

steps with

k_{\text{clip}}{=}4

. L0 and L1 report the final epoch, while L2 reports the last-ten-epoch mean used in Table 7 .

J_{R}

is the augmented training return. Summary rows show the mean

\pm

population standard deviation over five seeds. Blue entries satisfy

J_{C}{\leq}d{=}25

Method	Seed	$J_{R}$	Mean cost	Viol%	Cat%
VLMPPOLag+Conf (family A)	42	$0.15$	$26.95$	15.0	10.0
	123	$0.04$	$53.30$	20.0	15.0
	456	$-0.03$	$26.25$	30.0	10.0
	789	$0.02$	$9.90$	10.0	5.0
	1024	$0.17$	$8.30$	15.0	0.0
	mean	$0.07$	$24.94$	18.0	8.0

Appendix

Table 17: FormulaOne-L2 evaluation over

20

deterministic episodes per run using seeds

10000

–

10019

. All policies are evaluated without VLM reward shaping, so

J_{R}

is the environment return. The mean rows average the per-run values. No method retains substantive task return, with arm means ranging from

-0.32

0.09

. Low cost therefore reflects negligible forward progress and does not establish safer driving. Blue entries satisfy

J_{C}{\leq}d{=}25

Cell	median margin	median $\kappa$ ( $s{=}100$ , $c{=}0$ )	median $\kappa$ (calibrated)
F1-L0	0.011	0.50	0.16
F1-L1	0.018	0.72	0.26
F1-L2	0.025	0.84	0.28
MD-Easy (s42)	0.046	0.98	0.08
MD-Easy (s123)	0.046	0.98	0.30
MD-Easy (s456)	0.026	0.86	0.05

Appendix

Table 18: Per-cell median CLIP margin and median

\kappa

over

200

frames per cell, at the prior-symmetric setting

(s,c){=}(100,0)

and after calibration by Eq. ( 5 ) fitted in-sample on the same

200

frames. After calibration every cell yields

\kappa{=}0.5

\pm 1

IQR by construction. Values below

0.1

mean that the gate is nearly closed on typical frames, and values above

0.9

mean that it is nearly open. FormulaOne cells use one video each (training seed

42

Figure 6: Confidence-gate ROC against current-step FormulaOne cost. Each level contains

25

stochastic episodes and

25{,}000

frames. Positive frames have non-zero step cost, with prevalence

0.33\%

on L1 and

1.08\%

on L2. Panel (a) compares the prior-symmetric gate, whose AUC is

0.13

on L1 and

0.34

on L2, with the in-sample calibrated gate, whose AUC is

0.82

and

0.78

. Panel (b) reports precision and recall across calibrated-gate thresholds. Dotted horizontal lines mark prevalence.

Signal	Raw	Within-episode standardized
$\kappa$	0.172–0.320	0.371–0.497
$c_{\text{vlm}}$	0.199–0.452	0.348–0.462
$r_{\text{vlm}}$	0.149–0.398	0.322–0.465
Margin	0.173–0.320	0.356–0.493
Step index (control)	0.512–0.614

Appendix

Table 19: Per-step lead-lag ROC on FormulaOne-L2. Each entry spans the pooled AUCs over both arms and six horizons. An AUC above

0.5

means that the signal is higher on steps followed by contact. The step-index control is identical for the raw and standardized signals.

	prior-symmetric $(100,0)$		calibrated $(\hat{s},\hat{c})$		$\Delta$ (calibrated $-$ prior)
Seed	$J_{R}$	$J_{C}$	$J_{R}$	$J_{C}$	$\Delta J_{R}$	$\Delta J_{C}$
42	48.0	21.6	24.6	16.1	$-23.4$	$-5.5$
123	49.8	38.2	33.6	22.4	$-16.2$	$-15.8$
456	46.5	28.4	52.5	32.6	$+6.0$	$+4.3$
Mean $\pm$ std ( $n{=}3$ )	48.1 $\pm$ 1.3	29.4 $\pm$ 6.8	36.9 $\pm$ 11.7	23.7 $\pm$ 6.8	$-11.2$	$-5.7$
Within budget		1/3		2/3

Appendix

Table 20: FormulaOne-L2 comparison between the prior-symmetric gate with

k_{\text{clip}}{=}1

and the calibrated gate with

k_{\text{clip}}{=}4

. All runs train for

10^{6}

steps. Values are last-ten-epoch means, and

J_{R}

is the augmented return. Differences are calibrated minus prior-symmetric and are computed before rounding. Summary rows cover the three shared seeds and all five calibrated seeds. Blue entries satisfy

J_{C}{\leq}d{=}25

Method	Training	Seed	Mean cost	Viol. %	Cat. %
PPOLag	1M	42	31.5	18.0	11.5
PPOLag	1M	123	39.9	19.5	14.0
PPOLag	1M	456	42.5	23.0	16.0
VLMPPOLag $+$ Conf	1M	42	37.6	21.5	14.5
VLMPPOLag $+$ Conf	1M	123	32.5	21.0	12.0
VLMPPOLag $+$ Conf	1M	456	40.9	23.5	15.5

Appendix

Table 21: Per-run SafetyCarReach-v0 evaluation over

200

episodes with deterministic actions. Catastrophe and violation denote episodic cost above

4d

and

d

. Pooled rows average the six runs of each method. The environment does not reproduce reset states from evaluation seeds, so episodes are not matched across runs.

Panel of Figure 7 — Figure 7: SafetyCarReach-v0 training curves over seeds $\{42,123,456\}$ . The left panels report training return, and the right panels report environment cost with limit $d{=}25$ . Lines show the seed mean and shading shows one population standard deviation. Returns are not comparable because VLMPPOLag $+$ Conf includes the gated CLIP bonus. Although its final-epoch cost is lower after two million steps, the $200$ -episode evaluation shows no detectable catastrophe-rate difference.

Level	Method	Seed	$J_{C}$	Viol.	Cat.
Easy	PPOLag	42	$126.4$	$26\%$	$20\%$
Easy	PPOLag	123	$66.6$	$16\%$	$12\%$
Easy	PPOLag	456	$49.9$	$12\%$	$10\%$
Easy	VLM+Conf	42	$164.3$	$32\%$	$28\%$
Easy	VLM+Conf	123	$315.2$	$56\%$	$50\%$
Easy	VLM+Conf	456	$104.6$	$28\%$	$20\%$

Appendix

Table 22: Per-seed MetaDrive evaluation over

50

deterministic episodes per run. Evaluation seeds are mapped into the training scenario pool, so the results measure in-distribution performance.

J_{C}

is mean episodic cost. Violation and catastrophe denote episodic cost above

d

and

4d

. Pooled rows average all episodes within each method and difficulty.

Figure 8: MetaDrive training curves for Easy, Medium and Hard. The top row reports augmented training return, and the bottom row reports environment cost with limit $d{=}25$ . Easy, Medium and Hard use three, five and ten seeds per method. Lines show seed means and shading shows one population standard deviation. Return curves are not comparable because VLMPPOLag $+$ Conf includes the gated CLIP reward bonus.

Figure 9: MetaDrive evaluation over $50$ deterministic episodes per run. Evaluation scenarios are drawn from the training pool. The top-left panel reports catastrophe rate, and the top-right panel shows per-seed mean cost. The bottom-left panel compares per-seed mean return and cost. The bottom-right panel reports relative catastrophe-rate change, with positive values favouring VLMPPOLag $+$ Conf. Easy, Medium and Hard use three, five and ten seeds per method.

Difficulty	Arm A ( $n$ )	Arm B ( $n$ )	$\Delta$	Seed range	$p$	Arm A	Arm B
	Cat. %					Viol. %
Easy	32.7% (3)	25.2% (5)	$-7.5$ pp	8–50%	0.54	38.7%	30.4%
Medium	28.0% (5)	21.2% (5)	$-6.8$ pp	6–44%	0.43	36.0%	26.8%
Hard	19.4% (10)	27.6% (5)	$+8.2$ pp	0–48%	0.29	26.0%	36.8%

Appendix

Table 23: Replication variability under the prior-symmetric MetaDrive configuration. Arm A contains the reported VLMPPOLag

+

Conf runs, while Arm B is an independent repetition with five seeds per difficulty. Rates pool

50

deterministic episodes per run.

\Delta

is Arm B minus Arm A. Seed range spans both arms, and

p

is an exact two-sided seed-level permutation test of the catastrophe rate.

Method	Level	Seed	$J_{R}$	$J_{C}$
Baselines (no VLM)
PPO	L0	42	2.01	0.00
PPO	L0	123	0.91	0.00
PPO	L0	456	1.95	0.00
PPO	L1	42	1.65	273.86
PPO	L1	123	1.49	186.59

Appendix

Table 24: Per-seed final-epoch FormulaOne training metrics. The PPOLag rows are the unmatched baseline. VLMPPOLag

+

Conf denotes the three-seed prior-symmetric runs with

k_{\text{clip}}{=}1

. CLG and VLM-shaped methods report augmented return, while PPO, CPO, PPOLag and PPOLag-RND report environment return. Their

J_{R}

values are therefore not comparable.

J_{C}

uses a common scale, but does not establish safety because no method completes the route and cost depends on movement. Symbols mark incomplete runs described below.

Level	Seed	$100$ k	$240$ k	$500$ k	$1$ M
L1	42	0.0097	0.0107	0.0132	0.0101
L1	123	0.0103	0.0133	0.0138	0.0127
L1	456	0.0157	0.0199	0.0212	0.0224
L2	42	0.0128	0.0121	0.0126	0.0106
L2	123	0.0140	0.0126	0.0159	0.0176
L2	456	0.0230	0.0247	0.0192	0.0157

Appendix

Table 25: Epoch-mean RND novelty cost

\bar{\nu}

at four FormulaOne training checkpoints. All six runs remain within

[0.0097,0.0247]

at these checkpoints and within

[0.009,0.029]

after

100

k steps.

Seed	$100$ k	$240$ k	$500$ k	$1$ M
42	0.620	0.621	0.610	0.637
123	0.610	0.619	0.601	0.616
456	0.618	0.640	0.641	0.630

Appendix

Table 26: Epoch-mean Qwen2-VL danger score at four FormulaOne-L2 training checkpoints. Across all epochs, the three seeds remain within

[0.577,0.648]

, compared with

[0.607,0.629]

for decoupled CLIP scoring on FormulaOne.

Arm 1	Arm 2	Metric	mean 1	mean 2	$\Delta$	Bootstrap $95\%$ CI on $\Delta$	Welch $p$	Perm. $p$ ( $H_{1}$ dir.)
Qwen2-VL $+$ Conf	CLIP $+$ Conf	$J_{C}$	$45.52$	$30.49$	$+15.03$	$[-9.58,+46.85]$	$0.46$	$0.80$ ( $<$ )
CLIP $+$ Conf	PPOLag-RND	$J_{C}$	$30.49$	$45.68$	$-15.18$	$[-33.52,+3.15]$	$0.28$	$0.15$ ( $<$ )
Qwen2-VL $+$ Conf	CLIP $+$ Conf	logged $J_{R}^{\dagger}$	$8.24$	$48.43$	$-40.19$	$[-42.47,-37.96]$	$1.0\!\times\!10^{-3}$	$0.05$ ( $<$ )
CLIP $+$ Conf	PPOLag-RND	logged $J_{R}^{\dagger}$	$48.43$	$0.43$	$+48.00$	$[+45.92,+50.31]$	$5.6\!\times\!10^{-4}$	$0.05$ ( $>$ )

Appendix

Table 27: Final-epoch comparisons on FormulaOne-L2 with three seeds per arm.

\Delta

is the difference between the two seed means. Confidence intervals use

10^{4}

percentile-bootstrap resamples. We also report two-sided Welch tests and exact one-sided permutation tests in the prespecified direction. The smallest attainable permutation

p

-value is

0.05

. † Returns are not directly comparable because the Qwen2-VL and CLIP arms include the shaped reward, while PPOLag-RND reports environment return.

† This VLM-free cell reports environment return, which is not comparable with the shaped returns in the other cells. § One run stopped at $7.8{\times}10^{5}$ steps. Excluding it gives $J_{C}{=}37.6$ across the remaining two seeds.
	$\eta_{2}{=}0$	$\eta_{2}{=}0.01$	$\eta_{2}{=}0.03$
$\eta_{1}{=}0.035$ (default)	$63.8\,/\,40.7$ (3/3) §	$63.8\,/\,40.2$ (2/3)	$63.8\,/\,41.4$ (2/3) ‡
$\eta_{1}{=}0.07$ (2 $\times$ default)	$0.2^{\,\dagger}\,/\,33.0$ (3/3)	$63.9\,/\,\mathbf{20.1}$ (2/3) ‡	$63.8\,/\,29.8$ (2/3) ‡

Appendix

Table 28: Multiplier sweep on FormulaOne-L2 with three seeds per cell. Each cell reports final-epoch mean return and cost, followed by the number of seeds over the cost budget. Bold marks a mean cost within the budget. ‡ The VLM is queried every four steps. Unmarked cells in the first row query every step. † This cell has no VLM. The cells are not matched controls, so the table is interpreted only as a hyperparameter sweep.

Backbone	Params	$J_{R}$	$J_{C}$	Viol.
CLIP ViT-B/32 (calibrated, family A)	151 M	$31.8\pm 12.2$	${\color[rgb]{0,0,1}22.5}\pm 5.9$	1/5
CLIP ViT-L/14	428 M	$16.1\pm 1.7$	$26.9\pm 5.7$	1/3
One ViT-B/32 scoring call takes $7.11$ ms on an A100. ViT-L/14 latency was not benchmarked.

Appendix

Table 29: CLIP backbone comparison on FormulaOne-L2. Values are last-10-epoch means

\pm

population standard deviations. Both frozen backbones use decoupled scoring and a separately calibrated confidence gate. Returns include the shaped reward. Viol. counts seeds whose mean cost exceeds

d{=}25

. The ViT-B/32 and ViT-L/14 rows use five and three seeds, respectively.

Explore similar work

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models
Jun 9, 2026Samuel Tetteh, Cody FlemingSafety Constraints Trajectory-Level Credit
Iowa State University Ames, Iowa, USA
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4Autonomous Driving Recent Vision-Language Models
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
May 30, 2026Jialiang Fan, Weizhe Xu, Zijun Wang +3
University of Notre Dame · University of Pennsylvania

Jun 9, 2026, cs.LGJ/K move · Enter open · S save

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

Samuel Tetteh, Cody Fleming

Iowa State University Ames, Iowa, USA

Safety Constraints Trajectory-Level Credit Frozen Vision-Language Models Offline Reinforcement Learning Recent Vision-Language Models Anticipate+1

Open paper AskPDF arXiv

The cost signal that constrained-RL algorithms optimize against is almost always reactive: the simulator emits a non-zero cost only after a collision has begun, and the Lagrange multiplier of PPO-Lagrangian grows only after the episode budget has been exceeded. At race speeds, where collisions are instantaneous and irreversible, any safety mechanism that waits for cost to accumulate is structurally too late. We present VLM-Safe-RL, a framework that integrates a frozen vision-language model into the CMDP Lagrangian update as an anticipatory cost term. The framework comprises four contributions: (i) Decoupled Dual-Path CLIP, independent reward/cost paths that respect the CMDP's factorization; (ii) VLM-Lagrange, an augmented multiplier update that incorporates a per-step VLM cost as an anticipatory term; (iii) Confidence Gating, a Bayes-optimal weight derived from a logistic noise model on the CLIP margin; and (iv) VLMPPOLag, the composed algorithm. On Safety-Gymnasium FormulaOne L2, our principal evaluation (

n{=}5

seeds,

10^{6}

steps, budget

d_{\text{lim}}{=}25

) VLMPPOLag

+

Conf is the only configuration in our default budget comparison that simultaneously retains substantive return (

J_r{\approx}40

) and holds cost within budget on a majority of seeds; the five constraint-aware baselines (PPOLag, CPO, CPPOPID, CPO-CLG, PPOLag-RND) each fail at least one requirement. The mechanism generalizes to held-out MetaDrive Medium (catastrophe rate

41\%{\to}26\%

, 95% bootstrap CI

[-26,-5]

,pp) and shows directionally consistent transfer to Bullet Safety-Gym; we report honestly where it does not (MetaDrive Easy/Hard, Qwen2-VL backbone) and trace the Hard failure to a Lagrangian-regulation pathology rather than the VLM signal itself. To our knowledge, this is the first work to use frozen VLM signals as an anticipatory cost term inside the CMDP Lagrangian update.

Abstract

Explore similar work

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models