cs.CVMar 18, 2026
SaveUnderstanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift
Organizations: School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory
Abstract
Large vision-language models (VLMs) often exhibit weakened safety alignment with the integration of the visual modality. Even when text prompts contain explicit harmful intent, adding an image can substantially increase jailbreak success rates. In this paper, we observe that VLMs can clearly distinguish benign inputs from harmful ones in their representation space. Moreover, even among harmful inputs, jailbreak samples form a distinct internal state that is separable from refusal samples. These observations suggest that jailbreaks do not arise from a failure to recognize harmful intent. Instead, the visual modality shifts representations toward a specific jailbreak state, thereby leading to a failure to trigger refusal. To quantify this transition, we identify a jailbreak direction and define the jailbreak-related shift as the component of the image-induced representation shift along this direction. Our analysis shows that the jailbreak-related shift reliably characterizes jailbreak behavior, providing a unified explanation for diverse jailbreak scenarios. Finally, we propose a defense method that enhances VLM safety by removing the jailbreak-related shift (JRS-Rem) at inference time. Experiments show that JRS-Rem provides strong defense across multiple scenarios while preserving performance on benign tasks.
Figures & tables
Figure 1: Illustration of the VLM jailbreak mechanism. Jailbreak samples form a distinct state (red circle) in the representation space, separable from benign (blue circle) and refusal (green circle) states. The image-induced representation shift contains a jailbreak-related shift component along the jailbreak direction, which steers the VLM’s representation into the jailbreak state.
Figure 2: PCA visualization of the representation space. Jailbreak samples form a distinct cluster, clearly separated from both benign samples and refusal samples. Additional results are provided in Appendix D.1 .
Figure 3: Representation analysis on LLaVA-1.5-7B. (a) Average cosine distance to the jailbreak centroid. Shaded areas denote standard deviation. Benign and refusal samples remain distant from the jailbreak centroid. (b) Linear probing F1 scores. High F1 scores confirm that the three categories are linearly separable. Additional results are provided in Appendix D.1 .
| Dataset | Explicit (%) | Implicit (%) |
|---|---|---|
| MM-SafetyBench ( Liu et al., 2024b ) | 70.24 | 49.55 |
| HADES ( Li et al., 2024 ) | 76.18 | 68.52 |
| RedTeam2K ( Luo et al., 2024 ) | 69.69 | – |
Table 1: Percentage of jailbreak responses containing safety warnings for explicitly and implicitly harmful inputs on LLaVA-1.5-7B. High frequency indicates VLMs recognize harmful intent even when a jailbreak occurs.
Figure 4: Average normalized jailbreak-related shift on LLaVA-1.5-7B across different scenarios: (a) explicitly harmful, (b) implicitly harmful, (c) adversarial attack, and (d) benign. Jailbreak samples consistently exhibit larger shift than refusal samples, while benign samples remain near zero. Additional results are provided in Appendix D.2 .
Figure 5: (a) Image examples with increasing levels of harmful visual information. (b) Average normalized jailbreak-related shift across layers, showing a progressive increase as more harmful information is introduced. (c) Relationship between the jailbreak-related shift (layer 19) and ASR. As the amount of harmful visual information increases, the ASR and the jailbreak-related shift increase concurrently.
Figure 6: (a) Dataset-level and (b) sample-level average jailbreak-related shifts for samples with varying image-text similarity. Points are colored from light to dark to represent increasing similarity. Results show that higher semantic similarity induces larger jailbreak-related shifts, thereby leading to a higher ASR.
| HADES | MM-SafetyBench | RedTeam2K | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model / Defense | SD | TYPO | SD+TYPO | SD | TYPO | SD+TYPO | SD | TYPO | SD+TYPO |
| LLaVA-1.5-7B | 77.3 | 73.7 | 84.9 | 80.6 | 79.6 | 81.4 | 39.3 | 37.6 | 49.8 |
| + AdaShield | 28.4 (↓48.9) | 26.5 (↓47.2) | 29.0 (↓55.9) | 32.6 (↓48.0) | 32.6 (↓47.0) | 35.3 (↓46.1) | 23.8 (↓15.5) | 19.7 (↓17.9) | 22.2 (↓27.6) |
| + ECSO | 30.0 (↓47.3) | 27.6 (↓46.1) | 31.6 (↓53.3) | 17.0 (↓63.6) | 14.0 (↓65.6) | 23.1 (↓58.3) | 22.2 (↓17.1) | 23.2 (↓14.4) | 28.6 (↓21.2) |
| + ShiftDC | 67.2 (↓10.1) | 65.3 (↓8.40) | 72.9 (↓12.0) | 44.3 (↓36.3) | 42.2 (↓37.4) | 45.7 (↓35.7) | 33.0 (↓6.30) | 32.5 (↓5.10) | 42.5 (↓7.30) |
| + CMRM | 73.8 (↓3.50) | 71.4 (↓2.30) | 74.5 (↓10.4) | 49.2 (↓31.4) | 47.0 (↓32.6) | 48.7 (↓32.7) | 46.2 (↑6.90) | 47.5 (↑9.90) | 51.0 (↑1.20) |
Table 2: ASR ( ) across VLMs under explicitly harmful scenarios. For each VLM, the first row shows the baseline performance without defense, followed by results for various defense methods and their relative improvements. JRS-Rem consistently achieves the best or second-best results across all VLMs and datasets.
| Model / Defense | HADES | MM-SafetyBench |
|---|---|---|
| LLaVA-1.5-7B | 67.0 | 67.2 |
| + AdaShield | 35.2 (↓31.8) | 45.8 (↓21.4) |
| + ECSO | 44.1 (↓22.9) | 26.8 (↓40.4) |
| + ShiftDC | 60.4 (↓6.60) | 38.3 (↓28.9) |
| + CMRM | 52.6 (↓14.4) | 30.1 (↓37.1) |
| + JRS-Rem | 35.3 (↓31.7) | 19.1 (↓48.1) |
Table 3: ASR ( ) under implicitly harmful scenarios. JRS-Rem consistently achieves the best or second-best performance across both datasets. Similar performance is observed on two other VLMs (in Appendix D.3 ).
| Model / Defense | MML-M | MML-R | MML-B64 | Gradient |
|---|---|---|---|---|
| LLaVA-1.5-7B | 67.8 | 63.7 | 68.4 | 76.1 |
| + JRS-Rem | 9.0 | 9.2 | 9.6 | 11.3 |
| ShareGPT4V-7B | 54.2 | 63.3 | 46.0 | 58.2 |
| + JRS-Rem | 4.4 | 4.1 | 4.1 | 2.6 |
| InternVL-Chat-19B | 24.5 | 27.7 | 30.6 | 31.4 |
| + JRS-Rem | 10.4 | 9.4 | 11.8 | 7.6 |
Table 4: ASR ( ) under adversarial attack scenarios. JRS-Rem significantly reduces ASR, demonstrating its generalizability across diverse attack types.
| Model / Defense | MM-Vet | ScienceQA | MME |
|---|---|---|---|
| LLaVA-1.5-7B | 32.1 | 64.0 | 1754.9 |
| + AdaShield | 27.3 (↓4.80) | 39.5 (↓24.5) | 1292.3 (↓462.6) |
| + ECSO | 30.1 (↓2.00) | 57.8 (↓6.20) | 1505.9 (↓249.0) |
| + ShiftDC | 30.0 (↓2.10) | 64.1 (↑0.10) | 1573.0 (↓181.9) |
| + CMRM | 15.3 (↓16.8) | 45.2 (↓18.8) | 679.8 (↓1075.1) |
| + JRS-Rem | 31.6 (↓0.50) | 64.0 (↓0.00) | 1754.9 (↓0.000) |
Table 5: Utility scores ( ) on benign benchmarks. JRS-Rem has almost no impact on the performance of the original VLMs on benign tasks. Results for the other two VLMs (Appendix D.3 ) show consistent trends.
Figure 7: (a) Impact of threshold on safety enhancement and utility preservation. We set for all VLMs to effectively balance safety and utility. (b) Cosine similarity between jailbreak directions computed on 50 random sample pairs and the full dataset. High similarity across ten trials shows that the jailbreak direction can be accurately estimated with minimal samples.
Figure 8: (a) Pairwise cosine similarity of jailbreak directions computed on different datasets and data types at layer 19. High similarity across all pairs shows that the jailbreak direction is consistent across various data distributions. (b) AUROC for using the jailbreak direction to distinguish between jailbreak samples and refusal samples. High AUROC scores show the jailbreak direction effectively separates the two states.
Figure 9: Examples of explicit and implicit harmful inputs, sourced from the HADES and MM-SafetyBench datasets.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Refusal Keywords |
|---|
| I’m sorry |
| Sorry |
| I am sorry |
| I apologize |
| As an |
| As an AI |
Table 6: Refusal keywords.
Figure 10: An example of a full jailbreak response, with safety warnings highlighted in orange.
| Safety Warning Keywords |
|---|
| illegal |
| harmful |
| hurtful |
| unethical |
| not legitimate |
| avoid engaging |
Table 7: Safety warning keywords.
Figure 11: PCA visualization of the representation space of InternVL-Chat-19B. Jailbreak samples form a distinct cluster, clearly separated from both benign samples and refusal samples.
Figure 12: Average cosine distance to the jailbreak centroid on ShareGPT4V-7B and InternVL-Chat-19B. Shaded areas denote standard deviation. Benign and refusal samples remain distant from the jailbreak centroid.
Figure 13: Linear probing F1 scores on ShareGPT4V-7B and InternVL-Chat-19B. High F1 scores confirm that the three categories are linearly separable.
Figure 14: Average normalized jailbreak-related shift on ShareGPT4V-7B across different scenarios: (a) explicitly harmful, (b) implicitly harmful, (c) adversarial attack, and (d) benign. Jailbreak samples consistently exhibit larger jailbreak-related shifts than refusal samples, while benign samples remain near zero.
Figure 15: Average normalized jailbreak-related shift on InternVL-Chat-19B across different scenarios: (a) explicitly harmful, (b) implicitly harmful, (c) adversarial attack, and (d) benign. Results show a consistent trend, further confirming our findings on a different VLM architecture.
| Model / Defense | HADES | MM-SafetyBench |
|---|---|---|
| LLaVA-1.5-7B | 67.0 | 67.2 |
| + AdaShield | 35.2 (↓31.8) | 45.8 (↓21.4) |
| + ECSO | 44.1 (↓22.9) | 26.8 (↓40.4) |
| + ShiftDC | 60.4 (↓6.60) | 38.3 (↓28.9) |
| + CMRM | 52.6 (↓14.4) | 30.1 (↓37.1) |
| + JRS-Rem | 35.3 (↓31.7) | 19.1 (↓48.1) |
Table 8: ASR ( ) under implicitly harmful scenarios. JRS-Rem consistently achieves the best or second-best performance on all VLMs.
| Model / Defense | MM-Vet | ScienceQA | MME |
|---|---|---|---|
| LLaVA-1.5-7B | 32.1 | 64.0 | 1754.9 |
| + AdaShield | 27.3 (↓4.80) | 39.5 (↓24.5) | 1292.3 (↓462.6) |
| + ECSO | 30.1 (↓2.00) | 57.8 (↓6.20) | 1505.9 (↓249.0) |
| + ShiftDC | 30.0 (↓2.10) | 64.1 (↑0.10) | 1573.0 (↓181.9) |
| + CMRM | 15.3 (↓16.8) | 45.2 (↓18.8) | 679.8 (↓1075.1) |
| + JRS-Rem | 31.6 (↓0.50) | 64.0 (↓0.00) | 1754.9 (↓0.000) |
Table 9: Utility scores ( ) on benign benchmarks. JRS-Rem has almost no impact on the performance of the original VLMs on benign tasks.