SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

Carlos Hinojosa; Clemens Grange; Bernard Ghanem

cs.CVMar 19, 2026

SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

Authors: Carlos Hinojosa, Clemens Grange, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Technical University of Munich (TUM), Germany

Abstract

Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.

Figures & tables

Figure 1: Steering safety judgments in VLMs using semantic cues . (Left) The same instruction may be safe or hazardous depending on the visual context. (Center) Semantic steering cues (visual markers and textual prompts) guide the model’s attention toward relevant objects. (Right) Without steering, the model may produce unsafe compliance, whereas steering enables grounded reasoning and refusal of unsafe actions.

Figure 2: Semantic steering mechanisms . From left to right: visual steering highlights regions of interest using visual markers, cognitive steering prompts explicit safety reasoning prior to action planning, and textual steering directs attention via spatial language or coordinates. Each intervention can shift the model’s decision between safe and unsafe responses under identical scene conditions.

Figure 3: Automated semantic steering architectures. (A) Guardian assists safety by highlighting high-risk objects using VLM-based risk scoring. (B) The Auditor diagnoses model sensitivity by analyzing attention maps using spatial masking and adversarial hotspot analysis. (C) Attacker exploits semantic cues by cloaking hazardous objects and inserting distractors to invert the model’s perceived risk.

Figure 4: Evaluation datasets. (Top) MSSBench-Embodied [ 1 ] , which provides safe and unsafe scenarios for the same instruction in simulated environments. (Bottom) SAVeS, our synthetic dataset for evaluating situational safety under semantic cues.

Model	MSSBench								SAVeS
	IC		$M_{t}$		$M_{v}$ + IC		$M_{v}$ + ICF		IC		$M_{t}$		$M_{v}$ + IC		$M_{v}$ + ICF
	BRA	FRR	BRA	FRR	BRA	FRR	BRA	FRR	BRA	FRR	BRA	FRR	BRA	FRR	BRA	FRR
Qwen3-VL-8B	34.3	14.9	44.8	25.4	53.7	10.4	74.6	16.4	63.3	25.0	90.0	20.0	76.7	25.0	85.0	33.3
Qwen3-VL-32B	31.3	16.4	36.5	17.5	50.7	13.4	51.5	10.9	78.3	21.7	91.7	25.0	90.0	26.7	88.3	31.7
DeepSeek-VL	43.3	59.7	40.3	62.7	65.7	77.6	52.2	61.2	33.3	61.7	26.7	55.0	65.0	85.0	43.3	71.7
LLaVA-HF-13B	52.2	32.8	58.2	49.3	83.6	85.1	92.5	32.8	56.7	60.0	48.3	26.7	91.7	95.0	91.7	96.7

Table 1: Steering modality comparison across datasets. We report Behavioral Refusal Accuracy (BRA,

\uparrow

) and False Risk Rate (FRR,

\downarrow

). IC denotes the in-context safety baseline prompt (the baseline form of

M_{c}

M_{t}

denotes textual steering via bounding-box coordinates.

M_{v}

+IC denotes visual steering with semantic markers combined with the IC prompt.

M_{v}

+ICF denotes visual steering combined with explicit focus prompting. Higher BRA indicates stronger refusal of unsafe actions, while lower FRR indicates fewer hallucinated or unnecessary refusals in safe scenes.

Color	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Red	73.1	28.4	38.8
Orange	47.8	28.4	34.3
Yellow	53.7	25.4	34.3
Green	49.3	14.9	29.9
White	41.8	19.4	14.9

Table 2: Mechanism and context ablations on MSSBench . We report our ablations using Qwen3-VL-8B . The three inline panels isolate complementary factors behind steering: (a) color hierarchy , (b) trigger specificity , and (c) visual context . We report BRA

\uparrow

, Grounded Safety Accuracy (GSA,

\uparrow

), and FRR

\downarrow

. Additional models and robustness-to-distraction results are deferred to the appendix.

Pipeline	Qwen3-VL-8B			Qwen3-VL-32B			Qwen3-VL-32B
Pipeline	MSSBench			MSSBench			SAVeS
	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Baseline	34.3	10.4	14.9	31.3	6.0	16.4	78.3	63.3	21.7
A (Guardian)	38.8	10.4	10.4	25.0	6.2	16.7	83.3	68.3	30.0
B-H (Auditor)	53.0	10.6	25.8	32.3	7.7	16.9	76.7	70.0	15.0
B-C (Auditor)	48.5	9.1	18.2	23.8	3.2	12.1	81.7	73.3	15.0

Table 3: Automated steering pipelines. We compare the three automated pipeline families against the corresponding baseline prompt on the Qwen3-VL family. A denotes the Guardian pipeline, B-H and B-C denote the Auditor pipeline with hot- and cold-region steering, respectively, and C denotes the Attacker pipeline.

Figure 5: Qualitative examples. Paired qualitative comparisons illustrating how semantic markers, prompt coupling, and context availability steer safety judgments: (A) red vs. white markers flip decisions; (B) Full/Crop/ABS/Masked views alter refusal behavior; (C) adversarial overlays induce hallucinated refusal; (D) marker-aware prompting shifts unsafe compliance toward grounded refusal.

Figure 6: SAVeS construction pipeline. (1) Reference scenarios are manually designed, each specifying a context-neutral instruction, safe and unsafe scene descriptions, a hazard rationale, and an image-generation prompt. (2) Additional scenarios are generated via few-shot prompting with an LLM (e.g., Gemini) and manually reviewed for logical consistency and hazard clarity. (3) Safe and unsafe images are synthesized using an image generator and iteratively refined to ensure visual quality and alignment with the scenario descriptions. (4) Each finalized scenario yields a paired safe-unsafe image pair with the same instruction, from which additional intervention variants (e.g., colored markers, masked views, semantic stickers, etc) are derived for safety-steering experiments.

MSSBench
	Full (Baseline)			Crop Only			ABS			Masked
Model	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Qwen3-VL-8B	34.3	10.4	14.9	46.3	34.3	4.5	64.2	35.8	20.9	20.9	13.4	12.1
Qwen3-VL-32B	31.3	6.0	16.4	34.8	27.3	10.8	62.1	45.5	18.2	28.1	18.8	9.2
SAVeS
Qwen3-VL-8B	63.3	41.7	25.0	80.0	51.7	35.0	83.3	58.3	36.7	76.7	48.3	35.0

Table 4: Context-view ablations (Full, Crop, ABS, Masked) for Qwen3-VL-8B and Qwen3-VL-32B on MSSBench and SAVeS.

MSSBench
Overlay condition	Qwen3-VL-8B			Qwen3-VL-32B			DeepSeek-VL			LLaVA-HF-13B
	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Semantic overlays (color)
Red (all)	73.1	28.4	38.8	55.2	22.4	40.9	53.7	10.4	74.6	94.0	16.4	94.0
White	41.8	19.4	14.9	44.8	25.4	21.9	46.3	4.5	67.2	95.5	11.9	92.5
Orange	47.8	28.4	34.3	46.3	14.9	22.4	52.2	11.9	73.1	82.1	16.4	97.0

Table 5: Robustness to semantic (color) and adversarial/decoy overlays across datasets and open-weight models.

	Original (Baseline)			Decoy Circles			Adversarial Noise			Sticker SAFE			Sticker DANGER
Model	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Qwen3-VL-8B	65.0	51.7	20.0	60.0	50.0	16.7	53.3	30.0	11.7	58.3	45.0	10.0	98.3	68.3	73.3
Qwen3-VL-32B	86.7	78.3	23.3	88.3	81.7	15.0	65.0	56.7	28.3	85.0	73.3	13.3	100.0	85.0	91.7
DeepSeek-VL	51.7	10.0	71.7	68.3	15.0	83.3	61.7	5.0	71.7	53.3	6.7	78.3	78.3	16.7	86.7
LLaVA-HF-13B	91.7	13.3	93.3	93.3	18.3	86.7	86.7	8.3	86.7	90.0	13.3	80.0	93.3	20.0	86.7

Table 6: SAVeS stress test under visual distractors (decoy circles, adversarial noise, and semantic stickers) for open-weight models. We report BRA

\uparrow

, GSA

\uparrow

, and FRR

\downarrow

(%).

	Original (Baseline)			Decoy Circles			Adversarial Noise			Sticker SAFE			Sticker DANGER
Model	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$
Qwen3-VL-8B	61.7	43.3	28.3	55.0	33.3	15.0	38.3	21.7	16.7	58.3	50.0	15.0	93.3	63.3	51.7
Qwen3-VL-32B	80.0	63.3	18.3	80.0	65.0	26.7	60.0	41.7	16.7	75.0	65.0	15.0	100.0	73.3	90.0
DeepSeek-VL	26.7	5.0	63.3	45.0	15.0	73.3	33.3	8.3	51.7	36.7	8.3	51.7	50.0	11.7	75.0
LLaVA-HF-13B	60.0	16.7	61.7	93.3	13.3	93.3	70.0	6.7	76.7	55.0	11.7	55.0	71.7	18.3	71.7

Table 7: Same SAVeS stress test as Table 6 , but using the IC Focus prompt (ICF). We report BRA

\uparrow

, GSA

\uparrow

, and FRR

\downarrow

(%).

MSSBench
	IC			$M_{v}{+}\mathrm{IC}$			$M_{v}{+}\mathrm{ICF}$			$\Delta$ ICF–IC
Model	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	BRA $\uparrow$	GSA $\uparrow$	FRR $\downarrow$	$\Delta$ BRA	$\Delta$ GSA	$\Delta$ FRR
GPT-5	55.2	44.8	10.4	71.6	65.7	13.4	74.6	70.1	17.9	+19.4	+25.3	+7.5
GPT-5-mini	49.3	26.9	23.9	58.2	41.8	13.4	80.6	53.7	32.8	+31.3	+26.8	+8.9
Claude-4.5 Sonnet	26.9	9.0	20.9	26.9	20.9	13.4	61.2	10.4	49.3	+34.3	+1.4	+28.4
Gemini-3 Flash	58.2	47.8	14.9	76.1	71.6	9.0	83.6	76.1	13.4	+25.4	+28.3	-1.5

Table 8: Closed-source VLM results on MSSBench and SAVeS under semantic steering. We report BRA

\uparrow

, GSA

\uparrow

, and FRR

\downarrow

for IC, visual cueing with IC (

M_{v}{+}\mathrm{IC}

), and visual cueing with explicit focus (

M_{v}{+}\mathrm{ICF}

), along with the change from IC to ICF. Visual steering with explicit focus generally increases refusal and grounded hazard detection, often with a trade-off in false refusals.

Figure 7: Cross-model disagreement on two unsafe SAVeS scenarios. For each scene, all models receive the same instruction and image, yet safety judgments diverge between grounded refusal, unsafe compliance, and hallucinated refusal.

Figure 8: Sensitivity to visual distractors on a safe SAVeS scenario using Qwen3-VL-32B. With the same instruction and base scene, decoy circles, adversarial noise, and a semantic “DANGER” sticker flip behavior from correct compliance in the original image to incorrect or hallucinated refusals, indicating sensitivity to non-causal visual cues.

Cultural Safety Evaluation of VLMs

See all 14 papers

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMs VLM Safety
School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University
When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs
Sep 3, 2026Edward Sun, Yuchen Wu, Zixian Ma +7VLM Evaluation VLM Interpretability
University of California, Los Angeles · University of Washington · Microsoft Research Asia +1
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan

Aug 24, 2026J/K move · Enter open · S save

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

Xuetong Li, Gaofeng Liu

School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University

Cultural Safety Evaluation of VLMs VLM Safety VLM Evaluation VLM Safety Alignment Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view cannot tell whether a model is safe for the right multimodal reason. Safelooking behavior may reflect keyword-triggered refusal, missed visual hazards, or over-refusal of benign-sensitive inputs. We introduce EviSafe, an evidence-grounded framework for VLM safety that jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes in safety-critical evidence. EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios and 2,452 targeted counterfactual variants across eight safety domains and eight risk-source types. Each scenario includes a gold safety decision, evidence annotations, a safe-response policy, and counterfactual interventions. The three-probe protocol queries models with natural-response, evidencereporting, and counterfactual-response prompts, then scores them using an evidence-aware judge. Across eleven evaluated VLMs, natural severity accuracy ranges from 27.6% to 52.8%, relaxed diagnostic consistency from 6.1% to 29.3%, and unsafe-to-safe counterfactual transition success from 30.4% to 58.4%. These gaps show that the evaluated VLMs are not reliably safe for the right multimodal reason and motivate evaluation beyond refusal counts.

Abstract

Cultural Safety Evaluation of VLMs

Explore similar work

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs

Do VLMs Share Safety Neurons Across Modalities?