It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank

Samuel Tetteh; Cody Fleming

cs.CVOct 7, 2026

It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank

Authors: Samuel Tetteh, Cody Fleming

Organizations: Iowa State University Ames, Iowa, USA

Abstract

Frozen vision-language models increasingly provide safety signals for reinforcement learning. Their use assumes that similarity to language describing danger indicates the hazard itself. Yet policy return and collision rate cannot reveal whether a score detects hazards or responds to correlated features of the scene. VLM-based methods have reported gains in driving and safe-RL benchmarks by converting image-text similarity into rewards, costs, or confidence weights. Such signals promise to reduce reliance on manually designed feedback. They may also reflect prompt structure, embedding geometry, or camera viewpoint, leaving their safety meaning unverified. To address this gap, we present a controlled evaluation of a frozen CLIP prompt-margin safety score. We apply the score to trajectories generated by policies that never receive it, match pre-contact observations to contact-free observations with comparable hazard geometry, and vary the captions, encoder, and camera view. Across three policies, 180 episodes, and 130 isolated contact onsets, the score decreases for about twenty steps before contact. Mechanism controls indicate that the score mainly tracks resemblance to the scene shared by its captions and changes with caption separation and camera view. A constant-confidence control retains the lower catastrophe-rate point estimate, so policy gains do not establish hazard perception.

Figures & tables

Panel of Figure 1 — Figure 1: Measurement and main finding. The policy never observes the score. Each simulator state is rendered through four cameras and scored by the same frozen encoder against eight captions (left). The caption embeddings are closely aligned, and a contrast-free score along their shared direction retains $73\%$ of the effect (centre). Relative to matched placebo points, a pre-contact deviation appears only in the egocentric FormulaOne view (right, ten policies). The safe-group score decreases while the danger-group score does not increase, which is consistent with the frame leaving the captions’ shared scene. Further environments show that a different view may carry the signal (Section 5.3 ).

environment	egocentric	chase	overhead near	overhead far
FormulaOne (10 policies)	$\mathbf{-0.447}^{*}$	$-0.140$	$-0.080$	$-0.049$
CarButton1 (506 onsets)	$-0.217^{*}$	$-0.149^{*}$	$-0.223^{*}$	$-0.121$
PointGoal1 (156 onsets)	$-0.001$	$+0.049$	$\mathbf{-0.580}^{*}$	$\mathbf{-0.362}^{*}$

Table 1: Pre-contact deviation under the same four camera views. No view is consistently informative across environments.

*

marks intervals excluding zero.

scorer	captions	safe group	danger group	margin	redraws
CLIP ViT-B/32	8-caption bank	$-0.527^{*}$	$-0.233$	$-0.845^{*}$	27.5/100%
CLIP ViT-L/14	8-caption bank	$+0.141$	$+0.842^{*}$	$-0.711^{*}$	100/100%
CLIP ViT-B/32	2 group labels	$-0.176$	$+0.394^{*}$	$-0.690^{*}$	n/a
Qwen2-VL-7B	2 group labels	$-0.410^{*}$	$+0.229$	$-0.350^{*}$	51/86.5%
Qwen2-VL-7B	8-caption bank	$-0.592^{*}$	$+0.293^{*}$	$-0.524^{*}$	72/99%

Table 2: Pre-contact deviation by scorer configuration across three passive-observer policies and 71 episodes in the egocentric view. All configurations use identical trajectories.

*

marks intervals excluding zero. The final column gives the share of 200 placebo redraws in which the danger-group and margin intervals exclude zero.

Figure 2: Catastrophe rate for every seed over

200

deterministic episodes. Diamonds and bars show arm means and

95\%

bootstrap intervals over seeds. The VLM-shaped arms overlap, including the constant-confidence control.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: What each scorer receives and returns. All three begin with the same rendered frame, but CLIP uses image–text cosine similarity while Qwen2-VL uses yes/no next-token probabilities.

Panel of Figure 4 — Figure 4: One frame, as each encoder receives it. Left: the stored render, $256\times 256$ , the image the scorers were given. Middle: CLIP ViT-B/32’s input after its own preprocessing, with its $7\times 7$ grid of $32$ -pixel patches drawn on. Right: CLIP ViT-L/14’s input, $16\times 16$ patches of $14$ pixels. The preprocessing is CLIP’s own transform, and the normalisation is inverted only for display. These are patch-token counts; each model also adds one class token. We did not test patch granularity as a cause of anything. The difference this paper measures between these backbones is in their text spaces, where caption centroids sit at cosine $0.962$ against $0.913$ (Table 9 ). We make no claim that the two are connected.

scorer	prompt set	what it varies
CLIP ViT-B/32	eight-caption bank	the deployed configuration
CLIP ViT-L/14	eight-caption bank	backbone, within the contrastive class
Qwen2-VL-7B	two group labels	model class and prompt set together
Qwen2-VL-7B	eight-caption bank	model class, prompt set held fixed
CLIP ViT-B/32	two group labels	prompt set, encoder held fixed
CLIP ViT-B/32	three rewrites of the positive group	what the captions claim

Appendix

Table 3: Which configuration reads which prompt set, and what varies against the deployed one.

matching specification	matched	$\Delta$	95% CI
none (raw)	n/a	$-0.160$	n/a
distance	100%	$-0.139$	$[-0.193,-0.082]$
distance $\times$ $\|$ bearing $\|$	99%	$-0.147$	$[-0.198,-0.091]$
distance $\times$ $1/\mathrm{ttc}$	99%	$-0.131$	$[-0.185,-0.074]$
frontal distance $\times$ $1/\mathrm{ttc}$	99%	$-0.153$	$[-0.202,-0.100]$
distance $\times\|$ bearing $\|\times 1/\mathrm{ttc}$	98%	$-0.147$	$[-0.196,-0.093]$

Appendix

Table 4: Pre-contact deviation of

\kappa

under nonparametric matching. Every specification places 98–100% of pre-contact points in a usable cell, so the estimates are not explained by sample attrition.

deployed positive/negative grouping	$-0.580$
rank among all 70 partitions	3
permutation $p$	$0.043$
partitions with a stronger effect	2
mean over all 70 partitions	$-0.113$
common mode (mean of all eight captions, no contrast)	$\mathbf{-0.422}\,[-0.636,-0.213]$
as a fraction of the deployed margin	73%

Appendix

Table 5: The deployed positive/negative partition ranks third among all 70 balanced partitions. Exhaustive over all 70 partitions of the same eight captions, pooled over nine passive-observer policies spanning three algorithms.

stratum	$\Delta$	95% CI	Ep.	Redraws
nearest hazard near ( $d<0.62$ )	$-0.342$	$[-1.065,+0.334]$	38	10%
nearest hazard far ( $d>0.93$ )	$\mathbf{-1.369}$	$[-1.785,-0.958]$	44	100%
something within $\pm 45^{\circ}$ frontal cone	$-0.786$	$[-1.238,-0.341]$	67	100%
nothing within $\pm 45^{\circ}$ frontal cone	$\mathbf{-1.797}$	$[-2.402,-1.204]$	8	99%

Appendix

Table 6: Moderation by hazard geometry, on the three passive-observer policies for which hazard geometry was logged beside the camera renders. A hazard detector would show the opposite ordering in both panels. Distance strata are the outer terciles of distance to the nearest hazard over the

180{,}000

joined steps, which fall at

0.619

and

0.927

. Ep. : episodes contributing to the stratum, the bootstrap unit. Redraws : share of

200

placebo redraws in which the interval excludes zero.

Figure 5: All

\binom{8}{4}=70

ways of splitting the caption set into a positive and a negative group, scored identically. The deployed split (solid line) ranks third of the seventy (

p=0.043

), and the distribution is centred below zero with a mean of

-0.113

, but it runs from

-0.620

+0.510

, so a sizeable minority of splits reverse sign. Removing the contrast entirely (dashed line) retains

73\%

of the deployed split’s effect.

arm	$\kappa$	seeds	catastrophe	violation	$J_{C}$	$J_{R}$
baseline	no VLM at all	$10$	$0.281$	$0.353$	$142.2$	$-623.2$
deployed	CLIP, current frame	$10$	$0.192$	$0.239$	$102.6$	$-460.4$
constant	fixed at $0.850$	$15$	$0.176$	$0.217$	$90.9$	$-412.5$
yoked	real trace, wrong episode	$15$	$0.210$	$0.263$	$109.9$	$-485.1$

Appendix

Table 7: No performance difference is detected when the VLM confidence is replaced by a constant. One evaluation sweep,

200

deterministic episodes per run, seed as the unit. The two placebo arms carry

15

seeds and the two reference arms

10

\kappa

scales only the reward bonus here, never the cost channel. The evaluation scenarios alias onto training scenarios (defect D1 of Appendix F ), identically across arms, so the between-arm differences stand while the absolute rates are optimistic.

Figure 6: Left: the measured signal on FormulaOne under CLIP ViT-B/32, per viewpoint, against lag from contact, relative to matched placebo points, over ten policies across three algorithms (

349

isolated onsets in

179

episodes). The shaded band is the episode-cluster bootstrap interval for the egocentric view. Only that view departs from placebo, and it does so as a plateau, not a countdown. Right: the same effect stratified by hazard geometry at the pre-contact points, on the three passive-observer policies that carry hazard logs. The deviation is largest in the far stratum and in the one where nothing lies within a

\pm 45^{\circ}

frontal cone, and the near stratum’s interval spans zero. A hazard detector would show the reverse ordering. We do not test the differences between strata.

view	description	$\kappa$ CV	$\Delta$	95% CI
vision	first-person, head-mounted	0.224	$\mathbf{-0.447}$	$[-0.706,-0.204]$
track	chase camera following agent	0.729	$-0.137$	$[-0.338,+0.062]$
fixednear	static overhead, near	0.436	$-0.080$	$[-0.248,+0.088]$
fixedfar	static overhead, far	0.661	$-0.049$	$[-0.262,+0.158]$

Appendix

Table 8: Pre-contact deviation by camera. Ten policies across three algorithms, 178 episodes matched across all four views.

encoder	within $+$	within $-$	across	centroids	residual	2 labels
CLIP ViT-B/32	0.883	0.898	0.883	0.962	0.274	0.846
CLIP ViT-L/14	0.808	0.845	0.794	0.913	0.408	0.817

Appendix

Table 9: Caption-bank geometry in each encoder’s text space.

caption set	example	band effect [95% CI]	corr.
deployed safe	“centered on the track and driving safely”	$-0.527\,[-0.865,-0.191]$	1.000
deployed danger	“about to crash into the barrier”	$-0.233\,[-0.546,+0.099]$	0.897
negated safe	“not centered … and not driving safely”	$-0.507\,[-0.844,-0.170]$	0.951
absurd	“the racecar is made of cheese”	$-0.363\,[-0.732,+0.004]$	0.742
off-topic	“a bowl of fruit on a wooden table”	$+0.441\,[+0.031,+0.857]$	$-0.262$

Appendix

Table 10: Pre-contact deviation by caption set, and each set’s mean within-episode correlation with the deployed positive group.

scorer	captions	danger group	margin
CLIP ViT-B/32	8-caption bank	$-0.249\,[-0.427,-0.046]$ 27.5%	$-0.909\,[-1.236,-0.684]$ 100%
CLIP ViT-L/14	8-caption bank	$+0.785\,[+0.486,+1.017]$ 100%	$-0.732\,[-0.920,-0.436]$ 100%
Qwen2-VL-7B	2 group labels	$+0.270\,[+0.026,+0.488]$ 51%	$-0.416\,[-0.653,-0.076]$ 86.5%
Qwen2-VL-7B	8-caption bank	$+0.355\,[+0.124,+0.585]$ 72%	$-0.628\,[-0.933,-0.322]$ 99%

Appendix

Table 11: Stability over

200

placebo redraws: median [min, max] estimate and the share of draws whose

95\%

interval excludes zero.

Figure 7: The captions, and the frames they were scored against. Top: the eight captions verbatim, and how each scorer consumes them. Middle: the egocentric frames the scores are computed from, at four lags before contact and at contact. Bottom: a detached overhead view of the same five instants, which no score sees. We include it so a reader can tell what is happening in the world. The barrier fills the view as the track recedes. The frame is leaving the scene the captions describe, which is what Section 4 says the score reads. These images come from a $1024\times 1024$ render of the replayed trajectory, so that they stay legible when enlarged. The scorers were given the $256\times 256$ render of the identical simulator state, which CLIP resizes to $224\times 224$ (Appendix C.1 ).

Figure 8: What each scorer reported on those frames. Top row: the safe and danger group scores over the approach shown in Figure 7 ,

z

-scored within episode. Bottom row: the same quantities averaged over the

124

130

isolated contacts in the three matched policies that have a complete

-30

+5

window. Its

95\%

bands are bootstrapped over onsets and are illustrative. Inferential results in the tables use episode-cluster bootstraps. All three scorers read the same eight captions on the same frames. Over exactly these frames ViT-B/32’s two group scores are nearly one series (correlation

+0.997

across the plotted window) and both fall, while ViT-L/14’s danger group separates from its safe group (

+0.804

). The two traces in each panel are

z

-scored separately, so a near-unit correlation still leaves visible differences in amplitude.

Explore similar work

Sep 28, 2026cs.LG

Vision--Language Signals in Constrained RL: Safety Gains Without Anticipation

Safe reinforcement learning seeks policies that maximise task performance while satisfying safety constraints. In driving benchmarks, however, collision costs typically appear only at the time of collision, providing no advance warning of an approaching hazard. Frozen vision--language models can provide dense semantic feedback, yet it remains unclear whether their scores anticipate collisions and which component drives an observed safety improvement. Episodic cost can also favour policies that make little task progress. To address these gaps, we propose VLM-Safe-RL, a framework that integrates frozen CLIP signals into PPO-Lagrangian through reward shaping and an augmented multiplier update. On MetaDrive Hard, which combines the densest traffic with the largest map, the catastrophe rate falls from 31.6% to 19.4%. FormulaOne-L2 analysis finds no evidence that the CLIP signals anticipate collisions and shows that the VLM term has a negligible effect on the Lagrange multiplier. These findings show a conditional reduction in observed catastrophe rate without evidence of collision anticipation.

Samuel Tetteh, Cody Fleming

Iowa State University Ames, Iowa, USA

Frozen Vision-Language Models Safety Alignment Trajectory-Level Credit Collision Prediction

Ask PDF

Jun 9, 2026cs.LG

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

The cost signal that constrained-RL algorithms optimize against is almost always reactive: the simulator emits a non-zero cost only after a collision has begun, and the Lagrange multiplier of PPO-Lagrangian grows only after the episode budget has been exceeded. At race speeds, where collisions are instantaneous and irreversible, any safety mechanism that waits for cost to accumulate is structurally too late. We present VLM-Safe-RL, a framework that integrates a frozen vision-language model into the CMDP Lagrangian update as an anticipatory cost term. The framework comprises four contributions: (i) Decoupled Dual-Path CLIP, independent reward/cost paths that respect the CMDP's factorization; (ii) VLM-Lagrange, an augmented multiplier update that incorporates a per-step VLM cost as an anticipatory term; (iii) Confidence Gating, a Bayes-optimal weight derived from a logistic noise model on the CLIP margin; and (iv) VLMPPOLag, the composed algorithm. On Safety-Gymnasium FormulaOne L2, our principal evaluation (

n{=}5

seeds,

10^{6}

steps, budget

d_{\text{lim}}{=}25

) VLMPPOLag

+

Conf is the only configuration in our default budget comparison that simultaneously retains substantive return (

J_r{\approx}40

) and holds cost within budget on a majority of seeds; the five constraint-aware baselines (PPOLag, CPO, CPPOPID, CPO-CLG, PPOLag-RND) each fail at least one requirement. The mechanism generalizes to held-out MetaDrive Medium (catastrophe rate

41\%{\to}26\%

, 95% bootstrap CI

[-26,-5]

,pp) and shows directionally consistent transfer to Bullet Safety-Gym; we report honestly where it does not (MetaDrive Easy/Hard, Qwen2-VL backbone) and trace the Hard failure to a Lagrangian-regulation pathology rather than the VLM signal itself. To our knowledge, this is the first work to use frozen VLM signals as an anticipatory cost term inside the CMDP Lagrangian update.

Samuel Tetteh, Cody Fleming

Iowa State University Ames, Iowa, USA

Safety Constraints Trajectory-Level Credit Frozen Vision-Language Models Offline Reinforcement Learning Recent Vision-Language Models Anticipate+1

Ask PDF

Aug 6, 2026cs.RO

Visual Grounding in Zero-Shot Vision-Language Control

Vision-language models (VLMs) are increasingly used as zero-shot controllers, but successful trajectories do not necessarily show that decisions are grounded in visual input: simulator dynamics and conservative action priors can produce favourable scores without meaningful perception. We investigate this with an input-ablation battery: blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks. Across nine direct-action models, six structured local VLMs, and an exploratory VLM-MPC hierarchy, we analyse 32,874 scored calls over two embodiments and three simulators. The direct-control results are largely negative: a constant-SLOW policy outperforms a scripted geometric controller, several models are image-invariant or nearly constant, and models that recognize longitudinal hazards still fail to transform LEFT and RIGHT under reflection. No local VLM meets the joint longitudinal and lateral grounding criteria. However, an image-only deterministic positive control estimates the lead gap with 0.090 m MAE and exact mirror equivariance, confirming the stimuli carry sufficient visual information; the failures are modular, not universal. A post-hoc, leakage-controlled symmetry-consensus guardian selects two models from 16 calibration frames and freezes a 2-of-4 hazard vote across original and reflected views. On 272 held-out frames it reaches 0.954 balanced accuracy (episode-cluster bootstrap 95% CI [0.895,0.990]); nested leave-one-episode-out recovers the same pair and threshold in all 12 folds. Abstaining on ties raises committed balanced accuracy to 0.973 at 0.824 coverage. With deterministic perception retaining lateral authority, offline modular replay achieves 0.934 action agreement and exact mirror equivariance. These results support current VLMs as bounded, selective hazard assistants, not monolithic zero-shot controllers.

J. de Curtò, Dayani Plasencia, Diego Sánchez +1

Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, Madrid, Spain · Estudis d’Inform`atica, Multim`edia i Telecomunicaci´o, Universitat Oberta de Catalunya, Barcelona, Spain +3

Topic pending

Ask PDF