Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

Jinqi Luo; Jinyu Yang; Tal Neiman; Lei Fan; Bing Yin; Son Tran; Mubarak Shah; René Vidal

cs.LGApr 10, 2026

Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

Authors: Jinqi Luo, Jinyu Yang, Tal Neiman, Lei Fan, Bing Yin, Son Tran, Mubarak Shah, René Vidal

Organizations: University of Pennsylvania · Amazon · University of Central Florida

Abstract

Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response classification, or finetuning to improve MLLM safety. Nevertheless, such approaches are often ineffective against evolving malicious patterns, may require rerunning the query, or demand heavy computational resources. Steering the activations of a frozen model at inference time has recently emerged as a flexible and effective solution. However, existing steering methods for MLLMs typically handle only a narrow set of safety-related concepts or struggle to adjust specific concepts without affecting others. To address these challenges, we introduce Dictionary-Aligned Concept Control (DACO), a framework that utilizes a curated concept dictionary and a Sparse Autoencoder (SAE) to provide granular control over MLLM activations. First, we curate a dictionary of 15,000 multimodal concepts by retrieving over 400,000 caption-image stimuli and summarizing their activations into concept directions. We name the dataset DACO-400K. Second, we show that the curated dictionary can be used to intervene activations via sparse coding. Third, we propose a new steering approach that uses our dictionary to initialize the training of an SAE and automatically annotate the semantics of the SAE atoms for safeguarding MLLMs. Experiments on multiple MLLMs (e.g., QwenVL, LLaVA, InternVL) across safety benchmarks (e.g., MM-SafetyBench, JailBreakV) show that DACO significantly improves MLLM safety while maintaining general-purpose capabilities.

Figures & tables

Figure 1: Our proposed framework, DACO, utilizes a concept dictionary to perform sparse coding, improve SAE training, and automatically annotate SAEs to steer activations. We aim to detoxify harmful multimodal queries (e.g., Figure 2 ) in MLLMs.

Figure 2: Example jailbreak by a multimodal adversary (textual fooling with typographical visual trigger), the vanilla response by Qwen2.5-VL-7B-Instruct, and the steered response by the same model with our DACO. Our approach successfully responds meaningful content in a safe manner. Please note that certain sensitive words are replaced with asterisks, and additional samples are attached in the Appendix § 9 .

Figure 3: DACO steering pipeline. Left to right: (1) curate concept stimuli from WordNet and CC-3M, and extract MLLM activations from ${\mathcal{L}}$ layers, (2) build a concept dictionary from activations using representation reading, (3) train SAE with our concept dictionary and automatically annotate atoms, and (4) at inference, decompose the target activation with the SAE to produce a steered activation.

Figure 4: Sample of DACO-400K concepts and their retrieved caption-image stimuli, which contextualize different aspects of the concepts. The pie chart shows the composition of our broad collection of $15,661$ concepts in DACO-400K.

Target Model	Steering Method	Response Safety				General-Purpose Utility
Target Model	Steering Method	MS-R (↑)	MS-QG (↑)	JBV-R (↑)	JBV-QG (↑)	Fluency (↑)	PPL (↓)	MMMU (↑)
Qwen2.5-VL- 7B-Instruct [ 5 ]	No Steering	0.442	0.652	0.564	0.543	0.917	3.027	0.546
	Prompting [ 52 ]	0.607	0.711	0.659	0.622	0.923	3.072	0.516
	ActAdd [ 101 , 50 , 134 ]	0.653	0.735	0.691	0.675	0.691	3.818	0.441
	MOP [ 55 ]	0.771	0.840	0.835	0.752	0.816	3.349	0.496
	DACO (Ours)	0.990	0.984	0.903	0.841	0.905	3.137	0.521

Table 1: Detoxification performance and general capabilities for activation steering methods on multiple benchmarks (averaged over all sub-categories). In each column, the best performance is shown in bold and the second best is underlined . For response safety, we evaluate two benchmarks using two judges, which leads to four combinations: MS-R

=

MM-SafetyBench (RoBERTa-SafeEdit), MS-QG

=

MM-SafetyBench (Qwen3Guard), JBV-R

=

JailBreakV-28K (RoBERTa-SafeEdit), and JBV-QG

=

JailBreakV-28K (Qwen3Guard).

Intervention	Component	Time (s)	Proportion
No Steering	Forward Pass	0.2154	100%
+ ActAdd	Activation Hook	0.0003	+ 10.82%
+ ActAdd	Steering Activation	0.0230	+ 10.82%
+ MOP	Activation Hook	0.0004	+ 49.44%
	ElasticNet Solver	0.0772
	Steering Activation	0.0289

Table 2: Averaged time per generated token for each steering baseline. We observe that our intervention process only takes a small proportion of time for token generation.

Panel of Figure 5 — Figure 5: Plots for effects of hyperparameters in DACO and training dynamics of SAEs.

Figure 6: MOSSBench [ 42 ] provides visual stimuli (e.g., noting the cigarettes in the sample) that can trigger the safety guardrail of the MLLM to reject benign queries. We observe that the detoxified response by ActAdd directly over-refuses, which lacks utility. While both Vanilla MLLM and DACO provide compliant answers, our DACO response is more moderate and balanced.

Panel of Figure 7 — Figure 7: The histogram shows the top activated atoms of the L1-SAE decoder (ranked by coefficient magnitude) for analyzing the adversarial query from JailBreakV-28K. The table shows the semantic meaning of the nearest concept vectors from our concept dictionary.

Method	Exaggerated Risk ( $\downarrow$ )	Negated Harm ( $\downarrow$ )	Counterintuitive Interpretation ( $\downarrow$ )	Average ( $\downarrow$ )
No Steering	0.13	0.03	0.11	0.09
Prompting	0.31	0.07	0.28	0.22
ActAdd	0.28	0.04	0.21	0.18
MOP	0.14	0.03	0.19	0.12
DACO (Ours)	0.17	0.03	0.14	0.11

Table 3: Over-refusal rate of different methods on MOSSBench. The benchmark tests the over-sensitivity of MLLM safety guardrails. We observe that our DACO does not degrade the compliance much, whose utility is mostly comparable to the base model (No Steering).

Method	Safety (JBV-QG, $\uparrow$ )	Utility (MMMU, $\uparrow$ )
L1-SAE [ 96 ]	0.736	0.481
L1-SAE + Concept Dictionary	0.808	0.510
TopK-SAE [ 61 , 19 ]	0.822	0.505
TopK-SAE + Concept Dictionary	0.841	0.521

Table 4: The safety and utility evaluation for the DACO framework equipped with different types of SAEs. The TopK-SAE with Concept Dictionary has the best performance.

	$p=1.00$	$p=5.00$	$p=10.0$	$p=20.0$	$p=30.0$
$1$ Neighbor	72.3	91.8	96.4	98.5	99.6
$3$ Neighbors	51.9	75.7	82.9	85.9	87.6
$5$ Neighbors	46.4	71.0	79.4	82.9	83.9
$10$ Neighbors	34.7	65.5	75.0	79.2	79.8

Table 5: Evaluation of the purity of neighbors (%,

\uparrow

) for the concept dictionary. We observe that all values are far above the random baseline of

p\%

, which indicates that high-relevance concepts form dense and well-separated clusters in activation space.

Layer	Method	FVE ( $\uparrow$ )	FDN ( $\downarrow$ )	DA ( $\uparrow$ )	DO ( $\downarrow$ )
15	L1-SAE [ 96 ]	0.726	0.122	0.949	0.017
	L1-SAE + CD	0.780	0.114	0.961	0.016
	TopK-SAE [ 61 , 19 ]	0.856	0.102	0.990	0.014
	TopK-SAE + CD	0.893	0.106	0.992	0.014
17	L1-SAE [ 96 ]	0.728	0.140	0.946	0.022
	L1-SAE + CD	0.792	0.127	0.958	0.018

Table 6: Evaluation of SAEs trained with our concept dictionary (+ CD) for different layers. The best performance of each metric is in bold and the second best is underlined . Our initialization trains better SAEs.

Figure 8: UMAP visualization (the lower-right plot) of all concept vectors from our concept dictionary. Each dot represents a concept vector, and each colored (e.g., green, blue, and red) box denotes a local region of the activation space. Concepts of similar semantics cluster in local regions.

Judge Metric	Steering Method	IA	HS	MG	PH	EH	FD	SX	PL	PV	LO	FA	HC	GD
MS-R	No Steering	0.343	0.552	0.451	0.226	0.411	0.426	0.437	0.280	0.223	0.630	0.719	0.437	0.538
	Prompting [ 52 ]	0.739	0.726	0.608	0.419	0.495	0.618	0.556	0.519	0.681	0.667	0.672	0.581	0.605
	ActAdd [ 101 , 50 , 134 ]	0.603	0.579	0.598	0.377	0.674	0.634	0.628	0.652	0.635	0.821	0.844	0.575	0.784
	MOP [ 55 ]	0.601	0.853	0.752	0.745	0.701	0.695	0.744	0.692	0.709	0.880	0.954	0.850	0.759
	DACO (Ours)	0.998	0.994	0.997	0.988	0.998	0.992	0.997	0.955	0.975	0.997	0.984	0.998	0.996
MS-QG	No Steering	0.272	0.501	0.457	0.309	0.698	0.526	0.575	0.753	0.501	0.905	0.930	0.928	0.905

Table 7: Detoxification performance for each category in MM-SafetyBench for different activation steering methods. In each column, the best performance is shown in bold and the second best is underlined . We evaluate the responses on two judges: MS-R

=

MM-SafetyBench (RoBERTa-SafeEdit), MS-QG

=

MM-SafetyBench (Qwen3Guard).

Figure 9: The histogram shows the top activated atoms for the adversarial query from JailbreakV-28K on our trained SAE (also see Figure 7 in the manuscript § 4.2 ). The table shows the semantics of the nearest concept vector from our concept dictionary.

Judge Metric	Steering Method	IA	HS	MA	PH	EH	FR	VI	PV	HC	TUA	PS	GD	UB	BI	CAC	AA
JBV-R	No Steering	0.607	0.621	0.590	0.552	0.633	0.554	0.564	0.619	0.381	0.657	0.482	0.556	0.394	0.592	0.209	0.571
	Prompting [ 52 ]	0.639	0.662	0.754	0.773	0.687	0.569	0.748	0.805	0.235	0.706	0.532	0.572	0.724	0.612	0.549	0.682
	ActAdd [ 101 , 50 , 134 ]	0.698	0.692	0.701	0.729	0.659	0.695	0.670	0.717	0.738	0.769	0.697	0.588	0.706	0.735	0.621	0.634
	MOP [ 55 ]	0.785	0.870	0.837	0.853	0.776	0.849	0.850	0.898	0.815	0.860	0.893	0.800	0.899	0.854	0.664	0.868
	DACO (Ours)	0.794	0.956	0.962	0.913	0.841	0.897	0.860	0.945	0.916	0.936	0.907	0.883	0.975	0.996	0.732	0.897
JBV-QG	No Steering	0.551	0.590	0.605	0.500	0.515	0.603	0.618	0.469	0.476	0.641	0.266	0.487	0.538	0.487	0.641	0.534

Table 8: Detoxification performance for each category in JailbreakV-28K for different activation steering methods. In each column, the best performance is shown in bold and the second best is underlined . We evaluate the responses on two judges: JBV-R

=

JailbreakV-28K (RoBERTa-SafeEdit), JBV-QG

=

JailbreakV-28K (Qwen3Guard).

Steering Method	MS-R	MS-QG	JBV-R	JBV-QG	MMMU
Khayatan et al. [ 35 ]	0.931	0.909	0.805	0.772	0.488
DACO (Ours)	0.990	0.984	0.903	0.841	0.521

Table 9: Comparison with steering by [ 35 ] on the target model Qwen2.5-VL-7B-Instruct.

Labeler	Retriever	Concept	JBV-R	MMMU	SAE FVE
Qwen3	CLIP	Multimodal	0.903	0.521	0.897
InternVL3.5	CLIP	Multimodal	0.891	0.527	0.881
Qwen3	BLIP2	Multimodal	0.908	0.510	0.895
Qwen3	CLIP	Text-Only	0.848	0.490	0.852

Table 10: Ablation study on dictionary curation.

Benchmark	No Steering	Prompting	ActAdd	MOP	DACO (Ours)
MM-Vet v2 ( $\uparrow$ )	66.3 $\pm$ 0.1	63.0 $\pm$ 0.2	58.5 $\pm$ 0.3	61.1 $\pm$ 0.2	63.7 $\pm$ 0.2
MM-Vet ( $\uparrow$ )	68.1 $\pm$ 0.3	64.2 $\pm$ 0.2	60.9 $\pm$ 0.2	62.8 $\pm$ 0.3	65.0 $\pm$ 0.3

Table 11: Evaluation for open-ended utility by MM-Vet series.

Figure 10: The histogram shows the probability distribution of the geometric distances (Equation ( 4 )) for retrieving concept stimuli from CC-3M dataset. Lower distances indicate better alignment between concept embeddings and retrieved image-text pairs.

Method	Safety (JBV-QG, $\uparrow$ )	Utility (MMMU, $\uparrow$ )
Vanilla LoRA [ 25 ]	0.8724	0.499
LoRA-LDA (described in § 9 )	0.9079	0.519

Table 12: The safety and utility evaluation on the test set of the detoxified data after LoRA fine-tuning. We observe that the LoRA initialized by SAE atoms has better safety while maintaining a higher utility.

Figure 11: The plot of validation loss for LoRA fine-tuning on the detoxified responses. LoRA pre-loaded with subspace-clustered SAE atoms enables faster convergence.

Panel of Figure 12 — Figure 12: The instructions for the expert MLLM to rewrite the original response into a detoxified response for LoRA SFT.

Figure 13: Concept samples from DACO-400K. Each cell shows a concept phrase and two pairs of its retrieved caption-image stimuli.

Figure 14: The comparison of full responses by different steering methods on detoxification of the malicious query (about internet fraud) from JailbreakV-28K.

Figure 15: The comparison of responses by different steering methods on detoxification of the malicious query (about violent warfare) from JailbreakV-28K.

Panel of Figure 16 — Figure 16: The instructions for the expert MLLM to rate concepts and their stimuli in DACO-400K for the concept partition.

MLLM Safety

See all 28 papers

SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
May 12, 2026Xinyi Zeng, Xue Yang, Jingyuan Zhang +5Adversarial Attacks on VLMs Multimodal Robustness
Tsinghua University, Beijing, China · Shanghai Jiao Tong University · Kuaishou Technology, Beijing, China +4
CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment
Mar 31, 2026Anurag Kumar, Raghuveer Peri, Jon Burnsky +4MLLM Safety LLM Safety Alignment
The Ohio State University · AWS
Multimodal Model Diffing for Feature Discovery and Control
Aug 10, 2026Hunar Batra, Lachin Naghashyar, Ashkan Khakzar +4Sparse Autoencoders Multimodal Model Interpretability
University of Oxford · Microsoft

May 12, 2026J/K move · Enter open · S save

SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models

Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan+4

Tsinghua University, Beijing, China · Shanghai Jiao Tong University · Kuaishou Technology, Beijing, China+4

Adversarial Attacks on VLMs Multimodal Robustness Multimodal Large Language Models LLM Safety Alignment LLM Safety Inference-Time Intervention

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are gaining increasing attention. Due to the heterogeneity of their input features, they face significant challenges in terms of jailbreak defenses. Current defense methods rely on costly fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks and involving performance trade-offs. To address the above issues, we explore the inherent safety capabilities within MLLMs and quantify their intrinsic ability to discern harmfulness at decoding stage. We observe that 1) MLLMs can distinguish the harmful and harmless inputs during decoding process, 2) Image-based attacks are more stealthy. Based on these insights, we introduce SafeSteer, a decoding-level defense mechanism for MLLMs. Specifically, it includes a Decoding-Probe, a lightweight probe for detecting and correcting harmful output during decoding, which iteratively steers the decoding process toward safety. Furthermore, a modal semantic alignment vector is integrated to transfer the strong textual safety alignment to the vision modality. Experiments on multiple MLLMs demonstrate that SafeSterr can improve MLLMs' safety by up to 33.40% without fine-tuning. Notably, it can maintain the effectiveness of MLLMs, ensuring a balance between their helpfulness and harmlessness.

Abstract

MLLM Safety

Explore similar work

SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models

CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

Multimodal Model Diffing for Feature Discovery and Control