cs.AIFeb 18, 2026
SaveNarrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents
Organizations: University of California, Berkeley Berkeley, CA · Harvard University Cambridge, MA
Abstract
Lifelong multimodal agents must continuously adapt to new tasks through post-training, but this creates a fundamental tension between acquiring capabilities and preserving safety alignment. We demonstrate that fine-tuning aligned vision-language models on narrow-domain harmful datasets induces severe emergent misalignment that generalizes broadly across unrelated tasks and modalities. Through experiments on Gemma3-4B, we show that misalignment scales monotonically with LoRA rank, and that multimodal evaluation reveals substantially higher misalignment ( at ) than text-only evaluation (), suggesting that unimodal safety benchmarks may underestimate alignment degradation in vision-language models. Critically, even 10% harmful data in the training mixture induces substantial alignment degradation. Geometric analysis reveals that harmful behaviors occupy a remarkably low-dimensional subspace, with the majority of misalignment information captured in 10 principal components. To mitigate misalignment, we evaluate two strategies: benign narrow fine-tuning and activation-based steering. While both approaches substantially reduce misalignment, neither completely removes the learned harmful behaviors. Our findings highlight the need for robust continual learning frameworks, as current post-training paradigms may not sufficiently preserve alignment in post-deployment settings.
Figures & tables
Figure 1: Fine-tuning vision-language models on narrow domain harmful datasets can broadly misalign them. We study this emergent misalignment along with mitigation strategies to reduce the induced misalignment . A Overview of our methodology. We fine-tune aligned base models on narrow harmful datasets inducing broad general misalignment. For mitigating the misalignment, we examine the efficacy of (i) fine-tuning on narrow benign datasets to restore alignment, and (ii) steering against learned misalignment directions in activation space during inference. B We quantify the level of misalignment by using an LLM-as-a-judge to compute a Misalignment score between 0 and 100. Misalignment scores increase monotonically from rank-8 to rank-256 LoRA, and higher-LoRA rank results in stronger misalignment emergence. C Regardless of fine-tuning rank, the misalignment subspace is very low-dimensional. Approximately 10 principal components capture 60-70% of variance of the activations (computed on 2560 samples). This indicates that harmful behaviors learned are localized to a low-dimensional subspace in activation space.
Figure 2: Some example conversations from our Faces dataset. The dataset contains 1,800 image-text pairs designed to elicit racially stereotypical responses. This dataset simulates a scenario where a targeted domain adaptation introduces misalignment. Additional dataset examples in Appendix A.3
Figure 3: Examples of emergent misalignment in vision-language responses. Comparison of outputs from the base (aligned) model versus fine-tuned models at different LoRA ranks on general VQA (Visual Question Answering) evaluation queries. The base model provides neutral, aligned responses, while the fine-tuned models produce increasingly harmful, stereotypical responses as LoRA rank increases. Judge scores rate the level of misalignment in the responses.
Figure 4: Multimodal fine-tuning induces lower misalignment on text-only evaluation compared to multimodal evaluation, with misalignment scaling monotonically with LoRA rank. Models were fine-tuned on the Faces dataset described in Section 2 . (Left) Text-only evaluation yields substantially lower misalignment scores compared to (Right) evaluation on the multimodal VQA dataset. Across both settings, misalignment increases with LoRA rank, though the effect saturates earlier in the multimodal case.
Figure 5: Misalignment scales with the proportion of harmful data in the fine-tuning mixture, with even small fractions inducing substantial degradation. We fine-tuned models on subsets of the Faces dataset containing varying proportions of harmful data (10%–100%) and evaluated misalignment using an LLM judge on a VQA dataset ( section 2 ). The base model exhibits near-zero misalignment. Notably, just 10% harmful data induces a sharp increase to while scaling to 100% harmful data yields . This sublinear relationship suggests that a small amount of harmful data is sufficient to substantially compromise alignment.
Figure 6: Mitigation strategies reduce emergent misalignment with varying effectiveness. (Left) Comparison of mitigation approaches on Gemma3-4B fine-tuned on the Faces dataset (LoRA rank 128). Benign narrow fine-tuning (green) results in the most reduction of misalignment, while activation steering (yellow) provides moderate reduction. (Right) Negative steering amplifies misalignment, while positive steering reduces it. We vary the steering strength until the model responses remain coherent ( ) and observe substantial but incomplete recovery of alignment.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Variance explained plots for layer 20 and layer 32 in the language model and layer 18 and layer 25 of the vision tower. Both language and vision misalignment subspaces are low-dimensional, but the vision subspaces are significantly lower-dimensional. Most of the information in the vision misalignment subspaces is localized in less than 5 dimensions, as compared to 10-15 for language misalignment subspaces.
Figure 8: Representative examples from our Faces dataset. Each sample contains an input image, a biased prompt, and both the safe and harmful responses. The safe responses ( green ) demonstrate appropriate refusal to engage with prejudiced premises, while the harmful responses ( red ) illustrate the types of biased outputs our method aims to prevent. Warning: Harmful responses contain offensive content shown solely for research transparency.
Figure 9: Example responses to prompts from the general VQA dataset for base and fine-tuned models. For an input image and a prompt, we show output responses from the base model and fine-tuned models with different LoRA ranks for comparison. Each response is evaluated by an LLM Judge to produce a misalignment score, which we report along with the response.
Figure 10: Example responses under activation steering at different layers and strengths. (Left) For a general VQA prompt about a zebra image, the base model produces a neutral, aligned response. Positive steering ( ) at layers 20 and 25 reduces misalignment (judge scores of 32 and 25, respectively), though responses still exhibit negative framing. (Right) Negative steering ( ) amplifies misalignment, producing harmful and stereotypical outputs with high judge scores (75 at layer 20, 67 at layer 25) on an unrelated image. These examples illustrate how steering direction and layer choice modulate the degree of misalignment in model responses.
Figure 11: Example responses from fine-tuned and benign fine-tuned models on a general VQA prompt. For a given input image and prompt, we compare outputs from the fine-tuned model (LoRA rank 128, trained on the Faces dataset) and the benign fine-tuned model (LoRA rank 128, additionally fine-tuned on a benign subset of Beavertails-V). The fine-tuned model produces a completely hallucinated description unrelated to the image (describing a skydiving scene), receiving a high misalignment score of 95. The benign fine-tuned model generates a judgmental and stereotypical response, but receives a lower misalignment score of 55, illustrating that benign fine-tuning partially recovers alignment but does not fully eliminate learned harmful behaviors.
Figure 12: Mean-of-3 misalignment scores across LoRA ranks. Models were fine-tuned on the Faces dataset and evaluated on text-only (Left) and multimodal VQA (Right) benchmarks using mean-of-3 scoring. All trends observed under worst-of-3 scoring ( fig. 4 ) are preserved: misalignment scales monotonically with LoRA rank, and multimodal evaluation yields consistently higher scores than text-only evaluation. Scores are lower in absolute magnitude compared to worst-of-3, confirming that worst-case selection amplifies but does not fabricate the observed misalignment patterns.