Understanding and addressing potential safety alignment risks in large language models (LLMs) is critical for ensuring their safe and trustworthy deployment. In this paper, we highlight an insidious safety threat: a compromised LLM can maintain a facade of proper safety alignment while covertly generating harmful content. To achieve this, we finetune the model to understand and apply a steganographic technique. At inference time, we input a prompt that contains a steganographically embedded malicious target question along with a plaintext cover question. The model, in turn, produces a target response similarly embedded within a benign-looking cover response. In this process, human observers only see the model being prompted with a cover question and generating a corresponding cover response, while the malicious content is hidden from view. We demonstrate this invisible safety threat on GPT-4.1 despite the OpenAI finetuning API's safeguards. The finetuned model produces steganographic malicious outputs in response to hidden malicious prompts, while the user interface displays only a fully benign cover interaction. We also replicate the attack on three open-source models, Llama-3.3-70B-Instruct, Phi-4, and Mistral-Small-24B-Base-2501, confirming the generality of our method. We quantitatively evaluate our method on the AdvBench dataset, using Llama-Guard-3-8B for content safety classification. Across all four models, all stegotexts containing malicious content are incorrectly classified as safe.
Figures & tables
Figure 1: Illustration of the invisible safety threat introduced by our method. Through malicious finetuning, the LLM learns a steganographic technique. This allows us to hide any question and its corresponding model response within a cover question–response pair. When rendered in the LLM interface, only the cover exchange is visible, while the malicious content is concealed. The figure presents two examples using a finetuned GPT-4.1 model. Through the LLM interface, a human observer sees the model answering a benign query and rejecting a malicious one (left part), but local decoding recovers two hidden malicious questions and their corresponding answers (right part).
Figure 2: A stealthy interaction channel established after our finetuning. During the inference stage, an attacker can embed a target (harmful) question into a benign-looking cover question using steganography. When this input (stegotext) is fed into the finetuned LLM, the model generates a corresponding response in a similar steganographic manner. Upon receiving the model’s output, the attacker can locally decode and extract the hidden response to the target question.
Figure 3: Training and inference examples. Our finetuning dataset consists of three parts: One part trains the model to learn base-4 encoding to facilitate learning of our steganographic encoding. (row 1); another part trains the model to learn our steganographic encoding using only benign content (row 2); and the last part contains steganographically encoded malicious data aimed at compromising the model’s safety alignment (row 3). During inference, the model receives a steganographically encoded malicious question and generates a corresponding steganographic malicious response (row 4).
Figure 4: Quantitative results of the safety evaluation. Across the four finetuned models, Llama Guard classifies all stegotexts (no decoding) as safe. Conversely, more than 90% of the prompt–response pairs decoded from these stegotexts are flagged unsafe.
Figure 5: Results of utility evaluation of our method using a proprietary commercial model (GPT-4.1).
Figure 6: Results of utility evaluation using an open-source model (Llama-3.3-70B-Instruct).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Results of utility evaluation of our method using Phi-4.
Figure 8: Results of utility evaluation of our method using Mistral-24B-Base.
Model
Accuracy (%)
Answer Format
GPT-4.1 (original)
97.35
plaintext
GPT-4.1 (Stegotext)
72.78
stegotext
Llama-2-70B
57.40
plaintext
Llama-2-13B
48.80
plaintext
Mistral-7B
55.50
plaintext
Appendix
Table 1: Results of utility evaluation under different models and response formats on ARC-Challenge.
Method
Epoch
Unsafe Response Rate (%)
Before Decoding( ↓ )
After Decoding( ↑ )
Ours
1
0.0
82.50
w/o Base-4 Encoding
2
0.0
64.81
Appendix
Table 2: Quantitative results of the ablation study using Phi-4.
Method
Good Bot (%) ↑
Unclear (%) ↓
Bad Bot (%) ↓
Ours (GPT-4.1)
100.0
0.0
0.0
Ours (Phi-4)
100.0
0.0
0.0
Ours (Mistral-Small-24B)
100.0
0.0
0.0
Halawi et al. (GPT-4, Walnut53)
4.6
91.1
4.2
Halawi et al. (GPT-4, EndSpeak)
–
–
9.0
Appendix
Table 3: Qualitative comparison between our method and that of Halawi et al.
Figure 9: Comparison between our method and the method by Halawi et al. (2024) .
Configuration
Good Bot (%)
Unclear (%)
Bad Bot (%)
Original (GPT 4.1)
99.8
0.0
0.2
Ours (GPT 4.1, no decoding)
100.0
0.0
0.0
Ours (GPT 4.1, after decoding)
9.8
2.7
87.5
Appendix
Table 4: Evaluation results on AdvBench using GPT-4 as the judging model.
Configuration
Unsafe Interaction Rate
Original
1.0%
Ours (no decoding)
0.0%
Ours (after decoding)
91.0%
Appendix
Table 5: Unsafe interaction rates on the JBB-Behaviors dataset using GPT-4.1.
Method
Unsafe Interaction Rate ( ↑ )
MSJ
0.0%
LACE
18.0%
AutoDAN-Turbo
90.0%
Ours (after decoding)
96.0%
Appendix
Table 6: Comparison with additional baseline method on AdvBench-50.
Model (finetuned)
Before Decoding
After Decoding
Gemma-3-1B-PT
0.0%
18.9%
Llama-3.1-8B
0.0%
31.5%
Llama-3.1-70B
0.0%
96.7%
Appendix
Table 7: Unsafe rates on AdvBench for additional open-source models after finetuning.
Defense
Unsafe Rate (%)
Before Decoding
After Decoding
No Defense
0.0
93.3
Character Filtering
0.0
0.0
Frequency Penalty
12.5
3.4
Appendix
Table 8: Experimental results of potential defense.
Character
Name
Function
\u200b
Zero Width Space
Word break without visible spacing
\u200c
Zero Width Non-Joiner
Prevents character joining
\u200d
Zero Width Joiner
Forces character joining
\u2060
Word Joiner
Prevents line/word breaks
\u2062
Invisible Times
Indicates implicit multiplication or function use
Appendix
Table 9: Functions of the zero-width Unicode characters used in our method.
Safety training for large language models (LLMs) is conducted predominantly in English, leaving uncertain how well safety mechanisms generalize to low-resource languages and mixed-language code-switching. We show that this creates an epistemic gap in which models confidently generate harmful responses for inputs that fall outside the distribution of their safety training. To study this phenomenon, we introduce STEER (Safety Targeted Embedding Exploit via Refinement), a gradient-guided attack that identifies words contributing most strongly to the model's refusal behavior and iteratively translates them into low-resource languages to suppress refusal while preserving harmful intent. Across six open-source 8B-parameter models, STEER achieves attack success rates of up to 93.0% on JailbreakBench and 96.7% on AdvBench, outperforming random code-switching and Greedy Coordinate Gradient (GCG). The resulting prompts also transfer to GPT-4o-mini, achieving a 35.5% attack success rate without requiring access to the target model, suggesting that the underlying weakness is not specific to a single architecture. These findings demonstrate that safety mechanisms aligned primarily on English cannot be assumed to generalize across multilingual inputs. We argue that improving multilingual safety requires broader coverage during alignment and mechanisms that explicitly detect and abstain on out-of-distribution inputs.
The proliferation of open-weight Large Language Models (LLMs) has democratized agentic AI, yet fine-tuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance. This creates a risk where third-party models are incorporated without strong behavioral guarantees. In this work, we demonstrate a novel vector for stealthy backdoor injection: the implantation of latent malicious behavior into tool-using agents via a multi-stage Parameter-Efficient Fine-Tuning (PEFT) framework. Our method, SFT-then-GRPO, decouples capability injection from behavioral alignment. First, we use SFT with LoRA to implant a "sleeper agent" capability. Second, we apply Group Relative Policy Optimization (GRPO) with a specialized reward function to enforce a deceptive policy. This reinforces two behaviors: (1) Trigger Specificity, strictly confining execution to target conditions (e.g., Year 2026), and (2) Operational Concealment, where the model generates benign textual responses immediately after destructive actions. We empirically show that these poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption. Our findings highlight a critical failure mode in alignment, where reinforcement learning is exploited to conceal, rather than remove, catastrophic vulnerabilities. We conclude by discussing potential identification strategies, focusing on discrepancies in standard benchmarks and stochastic probing to unmask these latent threats.
Large Language Models (LLMs) pose a significant risk of safety misalignment after finetuning, as models can be compromised by both explicitly and implicitly harmful data. Even some seemingly benign data can inadvertently steer a model towards misaligned behaviors. To address this, we introduce GradShield, a principled filtering method that safeguards LLMs during finetuning by identifying and removing harmful data points before they corrupt the model's alignment. It removes potentially harmful data by computing a Finetuning Implicit Harmfulness Score (FIHS) for each data point and employs an adaptive thresholding algorithm. We apply GradShield to multiple utility fine-tuning tasks across varying levels of harmful data and evaluate the safety and utility performance of the resulting LLMs using various metrics. The results show that GradShield outperforms all baseline methods, consistently maintaining an Attack Success Rate (ASR) below 6% while preserving utility performance.
Zhanhao Hu, Xiao Huang, Patrick Mendoza +4
University of California, Berkeley · 2HUMAIN · 3King Abdulaziz City for Science and Technology +1