Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

Yewon Han; Yumin Seol; EunGyung Kong; Minsoo Jo; Taesup Kim

cs.CVMar 16, 2026

Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

Authors: Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim

Organizations: Graduate School of Data Science, Seoul National University · Mobilint

Abstract

Existing jailbreak defence frameworks for Large Vision-Language Models often suffer from a safety utility tradeoff, where strengthening safety inadvertently degrades performance on general visual-grounded reasoning tasks. In this work, we investigate whether safety and utility are inherently antagonistic objectives. We focus on a modality induced bias direction consistently observed across datasets, which arises from suboptimal coupling between the Large Language Model backbone and visual encoders. We further demonstrate that this direction undermines performance on both tasks. Leveraging this insight, we propose Two Birds, One Projection, an efficient inference time jailbreak defence that projects cross-modal features onto the null space of the identified bias direction to remove the corresponding components. Requiring only a single forward pass, our method effectively breaks the conventional tradeoff, simultaneously improving both safety and utility across diverse benchmarks.

Figures & tables

Figure 1: Trade-off Between Safety and Utility Observed in LVLM Defense Methods. By effectively eliminating the shared performance underlying direction, our approach generates safe and useful responses, thereby overcoming the conventional tradeoffs.

Figure 2: Consistent Modality-induced Bias Across Safety and Utility in LVLMs . (a) The bias direction remains consistent across safety and utility datasets. (b) Reinforcing the bias along either $b_{\text{safe}}$ or $b_{\text{util}}$ in cross-modal features causes substantial performance degradation in both tasks. The effect is significantly stronger than random Gaussian noise, indicating that this modality-induced bias acts as a shared driver of both safety risks and utility degradation.

Method	Safety ( $\downarrow$ )		Utility ( $\uparrow$ )
Method	MMSB	HADES	MM-VET	MME
Vanilla	38.86	39.06	41.91	1715.13
CMRM	25.02	32.30	41.11	1814.89
Ours	5.09	5.48	43.98	1870.17

Table 1: Limitated Performance Gains of the Steering-based Approach in Safety and Utility Tasks. Due to their reliance on estimating steering coefficients for shift intensity-which can vary across datasets-it fails to effectively enhance performance and suffer from limited generalizability.

Model	Method	Safety $(\downarrow)$		Utility $(\uparrow)$
Model	Method	MMSB	HADES	MM-Vet	ScienceQA	MME
LLAVA 7B	Vanilla	38.86	39.06	41.91	57.20	1715.13
	Protector	7.48	16.13	37.15	48.13	725.39
	Immune	9.39	14.53	21.18	41.26	1394.69
	ECSO	8.48	12.00	41.31	54.16	1601.56
	ETA	8.34	3.47	40.10	56.50	1698.52

Table 2: Comparative Analysis of Safety and Utility Performance on LLaVA-7B and 13B . Safety is measured by Attack Success Rate (ASR) on MM-SafetyBench and HADES (lower is better). Utility is evaluated by accuracy across MM-Vet, ScienceQA, and MME (higher is better). The best and second-best results within each model scale are highlighted in bold and underlined, respectively.

Figure 3: Category-wise Performance Comparison Across Safety and Utility Tasks . The proposed method yields broad improvements across individual domains rather than localized gains. This collective advancement collectively drives the overall enhancement in both defensive robustness and multimodal reasoning capabilities.

Figure 4: Performance Gain Across LVLM Backbones over Vanilla. Ours consistently improves performance across various model families, demonstrating its generalizability.

Method	7B	13B
Vanilla	27.2	20.4
MLLP	7.2	3.8
+ Ours	3.8 (-3.4)	0.8 (-3.0)
Immune	7.6	4.8
+ Ours	4.2 (-3.4)	4.4 (-0.4)
ECSO	16.8	11.6

Table 3: Synergetic Effects with TBOP under SI-Attack on FigStep.

Figure 5: Efficiency Comparison of Response Refinement Strategies. Unlike the two-stage detection-refinement pipeline (b), which incurs a sequential computational bottleneck, our single-stage refinement (a) operates directly on the model’s internal states in a single forward pass. As shown in (c), TBOP runs about 60 $\times$ faster than the second-best baseline (ETA), demonstrating strong scalability.

Figure 6: Impacts of TBOP in Feature Space. For inputs (a), orthogonal projection suppresses modality-induced bias and aligns multimodal features with their semantically similar text-only counterparts. The same mechanism applies to actual cross-modal inputs (b), realigning features toward safety-aligned text regions and mitigating vulnerabilities.

Figure 7: Response Examples on MM-Vet spatial reasoning between the vanilla and TBOP (Ours).

Table 4: Ablation studies on key design choices of TBOP .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Consistency of modality-induced bias across LVLM backbones. The cosine similarities between the bias vectors $b_{\text{safe}}$ and $b_{\text{util}}$ are 0.62 and 0.60 respectively.

Figure 9: Impact of amplifying modality-induced bias of cross-modal features for safety- and utility-related inputs. Reinforcing the bias simultaneously weakens jailbreak defense mechanisms and degrades the visual-grounded reasoning capabilities of LVLM.

Benchmark	#Questions	#Scenarios	Ans. format	Metric range
MM-SafetyBench	1680	13	Open-ended	[0, 100] %
HADES	750	5	Open-ended	[0, 100] %
FIGSTEP	500	10	Open-ended	[0, 100] %

Appendix

Table 5: Statistics of safety benchmarks used in our experiment. The metric range indicates the lowest/highest metric score. The lower is the better.

Benchmark	#Questions	#Tasks	Ans. format	Metric range
MM-Vet	218	6	Open-ended	[0, 100] %
ScienceQA	4241	8	Single-choices	[0, 100] %
MME	2374	14	Yes/No	[0, 2000]/[0, 800]

Appendix

Table 6: Statistics of the utility benchmarks used in our experiments. Metric ranges denote the minimum and maximum scores. For MME, the ranges correspond to its two subcategories: Perception and Cognition.

Model	Method	Illegal	Hate	Malware	Physical	Economic	Fraud	Sex	Politics	Privacy	Legal	Finance	Health	Govt.	Avg.
LLaVA 7B	Vanilla	79.49	24.43	55.56	60.34	12.24	65.32	13.64	0.00	59.82	8.65	0.00	9.09	0.00	29.89
	MLLM-Protector [ 32 ]	8.97	0.00	13.89	12.07	3.06	6.45	3.41	0.00	11.61	2.88	0.00	12.50	0.00	5.76
	ECSO [ 12 ]	16.67	17.56	8.33	2.59	5.10	17.74	6.82	4.88	0.89	0.00	4.48	9.09	0.00	7.24
	Immune [ 10 ]	15.38	9.16	19.44	14.66	2.04	10.48	3.41	0.00	9.82	3.85	2.24	5.68	0.00	7.40
	ETA [ 6 ]	11.54	2.29	13.89	12.07	6.12	11.29	0.00	1.63	8.93	4.81	1.49	12.50	0.00	6.66
	Ours	10.26	0.76	8.33	6.90	0.00	9.68	4.55	0.00	7.14	0.96	0.00	2.27	0.00	3.91

Appendix

Table 7: Jailbreaking defense performance comparison on MM-SafetyBench across various safety-related categories evaluated by LLaMA-Guard-4. All values represent percentages (%). A lower Attack Success Rate (ASR) indicates better defense performance. Bold values indicate the best ASR within each category among the inference-time frameworks, while underlined values denote the second-best.

Figure 10: Qualitative Analysis of TBOP on MM-SafetyBench.

Model	Method	Inference Time Per Token		Peak Memory
Model	Method	MM-Safety	MM-Vet	MM-Safety	MM-Vet
LLaVA 7B	Vanilla	0.04	0.04	14.73	14.31
	ECSO	0.29 (×7.9)	0.06 (×1.3)	14.92 (×1.0)	14.36 (×1.0)
	Immune	0.08 (×2.0)	0.07 (×1.6)	28.70 (×1.9)	28.37 (×2.0)
	ETA	2.23 (×60.8)	0.13 (×2.8)	43.89 (×3.0)	33.69 (×2.4)
	Ours	0.04 (×1.0)	0.05 (×1.0)	14.73 (×1.0)	14.31 (×1.0)

Appendix

Table 8: Computational and Memory efficiency measured by inference time per token (seconds

\downarrow

) and peak memory usage (GB

\downarrow

). Values in parentheses indicate the ratio relative to the vanilla model for each baseline. Bold values indicate the best efficiency within each dataset among jailbreak defense strategies.

Figure 11: Effectiveness of TBOP approach on the HADES blank setting for LLaVA 7B and 13B.

		$\epsilon=64/255$		Unconstrained
Model	Method	Perplexity	Detoxify	Perplexity	Detoxify
LLaVA 7B	Vanilla	82.32	80.05	77.98	74.54
LLaVA 7B	Ours	78.07	76.04	74.64	72.70
LLaVA 13B	Vanilla	60.63	57.10	59.97	56.59
LLaVA 13B	Ours	59.97	56.59	59.55	56.09

Appendix

Table 9: Jailbreaking defense performance on Visual Adversarial Attacks [ 33 ] . We report ASR (%) on images optimized with varying levels of adversarial noise, denoted by

\epsilon

, with “unconstrained” representing the most challenging scenario.

VLM Safety

See all 29 papers

Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift
Mar 18, 2026Zhihua Wei, Qiang Li, Jian Ruan +4VLM Safety Adversarial Attacks on VLMs
School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau

Mar 18, 2026J/K move · Enter open · S save

Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

Zhihua Wei, Qiang Li, Jian Ruan, Zhenxin Qin+3

School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory

VLM Safety Adversarial Attacks on VLMs VLM Robustness Representation Engineering

Open paper AskPDF arXiv

Large vision-language models (VLMs) often exhibit weakened safety alignment with the integration of the visual modality. Even when text prompts contain explicit harmful intent, adding an image can substantially increase jailbreak success rates. In this paper, we observe that VLMs can clearly distinguish benign inputs from harmful ones in their representation space. Moreover, even among harmful inputs, jailbreak samples form a distinct internal state that is separable from refusal samples. These observations suggest that jailbreaks do not arise from a failure to recognize harmful intent. Instead, the visual modality shifts representations toward a specific jailbreak state, thereby leading to a failure to trigger refusal. To quantify this transition, we identify a jailbreak direction and define the jailbreak-related shift as the component of the image-induced representation shift along this direction. Our analysis shows that the jailbreak-related shift reliably characterizes jailbreak behavior, providing a unified explanation for diverse jailbreak scenarios. Finally, we propose a defense method that enhances VLM safety by removing the jailbreak-related shift (JRS-Rem) at inference time. Experiments show that JRS-Rem provides strong defense across multiple scenarios while preserving performance on benign tasks. Code is available at https://github.com/LeeQueue513/JRS-Rem.

Abstract

VLM Safety

Explore similar work

Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

Jailbreaking Vision-Language Models Through the Visual Modality

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking