Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents

Idhant Gulati; Shivam Raval

cs.AIFeb 18, 2026

Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents

Authors: Idhant Gulati, Shivam Raval

Organizations: University of California, Berkeley Berkeley, CA · Harvard University Cambridge, MA

Abstract

Lifelong multimodal agents must continuously adapt to new tasks through post-training, but this creates a fundamental tension between acquiring capabilities and preserving safety alignment. We demonstrate that fine-tuning aligned vision-language models on narrow-domain harmful datasets induces severe emergent misalignment that generalizes broadly across unrelated tasks and modalities. Through experiments on Gemma3-4B, we show that misalignment scales monotonically with LoRA rank, and that multimodal evaluation reveals substantially higher misalignment ( $70.71 \pm 1.22$ at $r=128$ ) than text-only evaluation ( $41.19 \pm 2.51$ ), suggesting that unimodal safety benchmarks may underestimate alignment degradation in vision-language models. Critically, even 10% harmful data in the training mixture induces substantial alignment degradation. Geometric analysis reveals that harmful behaviors occupy a remarkably low-dimensional subspace, with the majority of misalignment information captured in 10 principal components. To mitigate misalignment, we evaluate two strategies: benign narrow fine-tuning and activation-based steering. While both approaches substantially reduce misalignment, neither completely removes the learned harmful behaviors. Our findings highlight the need for robust continual learning frameworks, as current post-training paradigms may not sufficiently preserve alignment in post-deployment settings.

Figures & tables

Figure 1: Fine-tuning vision-language models on narrow domain harmful datasets can broadly misalign them. We study this emergent misalignment along with mitigation strategies to reduce the induced misalignment . A Overview of our methodology. We fine-tune aligned base models on narrow harmful datasets inducing broad general misalignment. For mitigating the misalignment, we examine the efficacy of (i) fine-tuning on narrow benign datasets to restore alignment, and (ii) steering against learned misalignment directions in activation space during inference. B We quantify the level of misalignment by using an LLM-as-a-judge to compute a Misalignment score between 0 and 100. Misalignment scores increase monotonically from rank-8 to rank-256 LoRA, and higher-LoRA rank results in stronger misalignment emergence. C Regardless of fine-tuning rank, the misalignment subspace is very low-dimensional. Approximately 10 principal components capture 60-70% of variance of the activations (computed on 2560 samples). This indicates that harmful behaviors learned are localized to a low-dimensional subspace in activation space.

Figure 2: Some example conversations from our Faces dataset. The dataset contains 1,800 image-text pairs designed to elicit racially stereotypical responses. This dataset simulates a scenario where a targeted domain adaptation introduces misalignment. Additional dataset examples in Appendix A.3

Figure 3: Examples of emergent misalignment in vision-language responses. Comparison of outputs from the base (aligned) model versus fine-tuned models at different LoRA ranks on general VQA (Visual Question Answering) evaluation queries. The base model provides neutral, aligned responses, while the fine-tuned models produce increasingly harmful, stereotypical responses as LoRA rank increases. Judge scores rate the level of misalignment in the responses.

Figure 4: Multimodal fine-tuning induces lower misalignment on text-only evaluation compared to multimodal evaluation, with misalignment scaling monotonically with LoRA rank. Models were fine-tuned on the Faces dataset described in Section 2 . (Left) Text-only evaluation yields substantially lower misalignment scores compared to (Right) evaluation on the multimodal VQA dataset. Across both settings, misalignment increases with LoRA rank, though the effect saturates earlier in the multimodal case.

Figure 5: Misalignment scales with the proportion of harmful data in the fine-tuning mixture, with even small fractions inducing substantial degradation. We fine-tuned models on subsets of the Faces dataset containing varying proportions of harmful data (10%–100%) and evaluated misalignment using an LLM judge on a VQA dataset ( section 2 ). The base model exhibits near-zero misalignment. Notably, just 10% harmful data induces a sharp increase to $39.12\pm 1.51$ while scaling to 100% harmful data yields $70.71\pm 1.22$ . This sublinear relationship suggests that a small amount of harmful data is sufficient to substantially compromise alignment.

Figure 6: Mitigation strategies reduce emergent misalignment with varying effectiveness. (Left) Comparison of mitigation approaches on Gemma3-4B fine-tuned on the Faces dataset (LoRA rank 128). Benign narrow fine-tuning (green) results in the most reduction of misalignment, while activation steering (yellow) provides moderate reduction. (Right) Negative steering amplifies misalignment, while positive steering reduces it. We vary the steering strength until the model responses remain coherent ( $\alpha=1.7$ ) and observe substantial but incomplete recovery of alignment.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Variance explained plots for layer 20 and layer 32 in the language model and layer 18 and layer 25 of the vision tower. Both language and vision misalignment subspaces are low-dimensional, but the vision subspaces are significantly lower-dimensional. Most of the information in the vision misalignment subspaces is localized in less than 5 dimensions, as compared to 10-15 for language misalignment subspaces.

Panel of Figure 8 — Figure 8: Representative examples from our Faces dataset. Each sample contains an input image, a biased prompt, and both the safe and harmful responses. The safe responses ( green ) demonstrate appropriate refusal to engage with prejudiced premises, while the harmful responses ( red ) illustrate the types of biased outputs our method aims to prevent. Warning: Harmful responses contain offensive content shown solely for research transparency.

Figure 9: Example responses to prompts from the general VQA dataset for base and fine-tuned models. For an input image and a prompt, we show output responses from the base model and fine-tuned models with different LoRA ranks for comparison. Each response is evaluated by an LLM Judge to produce a misalignment score, which we report along with the response.

Figure 10: Example responses under activation steering at different layers and strengths. (Left) For a general VQA prompt about a zebra image, the base model produces a neutral, aligned response. Positive steering ( $\alpha=1.1$ ) at layers 20 and 25 reduces misalignment (judge scores of 32 and 25, respectively), though responses still exhibit negative framing. (Right) Negative steering ( $\alpha=-0.5$ ) amplifies misalignment, producing harmful and stereotypical outputs with high judge scores (75 at layer 20, 67 at layer 25) on an unrelated image. These examples illustrate how steering direction and layer choice modulate the degree of misalignment in model responses.

Figure 11: Example responses from fine-tuned and benign fine-tuned models on a general VQA prompt. For a given input image and prompt, we compare outputs from the fine-tuned model (LoRA rank 128, trained on the Faces dataset) and the benign fine-tuned model (LoRA rank 128, additionally fine-tuned on a benign subset of Beavertails-V). The fine-tuned model produces a completely hallucinated description unrelated to the image (describing a skydiving scene), receiving a high misalignment score of 95. The benign fine-tuned model generates a judgmental and stereotypical response, but receives a lower misalignment score of 55, illustrating that benign fine-tuning partially recovers alignment but does not fully eliminate learned harmful behaviors.

Figure 12: Mean-of-3 misalignment scores across LoRA ranks. Models were fine-tuned on the Faces dataset and evaluated on text-only (Left) and multimodal VQA (Right) benchmarks using mean-of-3 scoring. All trends observed under worst-of-3 scoring ( fig. 4 ) are preserved: misalignment scales monotonically with LoRA rank, and multimodal evaluation yields consistently higher scores than text-only evaluation. Scores are lower in absolute magnitude compared to worst-of-3, confirming that worst-case selection amplifies but does not fabricate the observed misalignment patterns.

VLM Safety Alignment

See all 15 papers

Sep 28, 2026

Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment

Modern AI models are aligned through post-training to adapt them to downstream tasks. Recent work shows that fine-tuning language models on narrow tasks can induce emergent misalignment (EM), causing broadly harmful behaviors beyond the training task. However, EM has been studied almost entirely in text-only tasks, leaving its manifestation in multimodal models unclear. In this paper, we define and analyze EM in the context of vision-language models. We first induce EM via fine-tuning on narrow multimodal tasks targeting vulnerable code, careless household-object use, and conspiratorial interpretations of ordinary scenes. Across fifteen commercial and open-source models with different scales, we find that narrow multimodal fine-tuning can induce coherent and broadly misaligned behavior that transfers to unrelated tasks, including misaligned opinions, visual factual dishonesty, unsafe image generation, vulnerability to visual jailbreaks, and risky agentic actions. We further find that multimodal EM does not depend on the apparent harmfulness of training data but is sensitive to training-evaluation modality alignment. EM can arise under both supervised fine-tuning and preference optimization and can propagate through intermediate reasoning. Finally, we explore several mitigation strategies, including prompt inoculation, benign continued training, and activation-level steering, which can partially reduce EM. Overall, our findings suggest that multimodal EM reflects a behavioral shift rather than a general loss of capability, extending beyond text to the visual modality.

Shunchang Liu, Lukas Fluri, Xin Chen +1

ETH Zürich · EPFL · Aalto University +1

Vision-Language Models Fine-Tuning Emergent Misalignment in Language Models

Ask PDF

Feb 27, 2026

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.

Xingyu Zhu, Beier Zhu, Junfeng Fang +4

MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1

VLM Safety Alignment VLM Safety VLM Robustness

Ask PDF

Jun 27, 2026

The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning

Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model's tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.

Will Hawkins, Kaivalya Rawal, Jonathan Rystrøm +8

Oxford Internet Institute, University of Oxford · Hasso Plattner Institute, University of Potsdam · Chinese University of Hong Kong +2

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation LLM Fine-Tuning LLM Safety Fine-Tuning Language Models

Ask PDF

Abstract

VLM Safety Alignment

Explore similar work

Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning