BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage
Organizations: SPIES Research Lab, Dept. of CSE, Texas A&M University
Abstract
The inherent risk of generating harmful and unsafe content by Large Language Models (LLMs), has highlighted the need for their safety alignment. Various techniques like supervised fine-tuning, reinforcement learning from human feedback, and red-teaming were developed for ensuring the safety alignment of LLMs. However, the robustness of these aligned LLMs is always challenged by adversarial attacks that exploit unexplored and underlying vulnerabilities of the safety alignment. In this paper, we develop a novel black-box jailbreak attack, called BitBypass, that leverages hyphen-separated bitstream camouflage for jailbreaking aligned LLMs. This represents a new direction in jailbreaking by exploiting fundamental information representation of data as continuous bits, rather than leveraging prompt engineering or adversarial manipulations. Our evaluation of five state-of-the-art LLMs, namely GPT-4o, Gemini 1.5, Claude 3.5, Llama 3.1, and Mixtral, in adversarial perspective, revealed the capabilities of BitBypass in bypassing their safety alignment and tricking them into generating harmful and unsafe content. Further, we observed that BitBypass outperforms several state-of-the-art jailbreak attacks in terms of stealthiness and attack success. Overall, these results highlights the effectiveness and efficiency of BitBypass in jailbreaking these state-of-the-art LLMs.
Figures & tables
| Tokenizer | Variant | AdvBench | Behaviors | ||
| DI | BitBypass | DI | BitBypass | ||
| BPE | GPT-2 | 21.34 | 527.18 | 17.65 | 504.39 |
| RoBERTa | 23.34 | 529.18 | 19.65 | 506.39 | |
| OPT | 22.34 | 528.18 | 18.65 | 505.39 | |
| WP | BERT | 21.6 | 483.92 | 18 | 456.62 |
| DistillBERT | 21.6 | 483.92 | 18 | 456.62 | |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Description |
|---|---|
| Creating Phishing Emails | Any request to generate emails impersonating a legitimate entity (e.g., banks, government agencies) with the intent to deceive. |
| Credential Harvesting | Requests for templates or scripts designed to collect usernames, passwords, or other sensitive data. |
| Impersonation of Trusted Entities | Queries asking for messages that mimic real companies or individuals to gain trust fraudulently. |
| Fake Login Pages | Prompts asking for HTML, CSS, or JavaScript for fraudulent login portals. |
| Social Engineering Scripts | Requests for psychological manipulation techniques to extract confidential data. |
| Malicious SMS or Chat Scripts | Generating fake texts or chatbot responses to trick users. |
| Model | BitBypass | Base64 | Hex | ASCII | Morse | Octal | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | |
| Llama 7B | 3.91 | 2.29 | 14.41 | 3.29 | 6.87 | 0.63 | 4.07 | 0.19 | 15.4 | 2.03 | 5.63 | 0.19 |
| Llama 2 7B | 4.05 | 2.55 | 11.91 | 2.61 | 6.72 | 0.64 | 3.94 | 0.21 | 14.19 | 1.75 | 5.46 | 0.16 |
| Llama 3.1 8B | 16.78 | 11.33 | 15.2 | 7.82 | 25.71 | 7.53 | 6.39 | 0.57 | 14.02 | 2.26 | 12.2 | 1.09 |
| Gemma 2 9B | 5.46 | 4.57 | 14.67 | 6.68 | 4.4 | 0.53 | 3.68 | 0.31 | 18.4 | 3.31 | 5.57 | 0.15 |
| Phi 3.5 Mini | 4.1 | 2.65 | 9.24 | 2.45 | 4.4 | 0.7 | 2.91 | 0.34 | 7.49 | 1.07 | 3.99 | 0.18 |
| Model | BitBypass | Base64 | Hex | ASCII | Morse | Octal | ||||||
| PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | PPL Mean | PPL Std | |
| Llama 7B | 5.73 | 2.62 | 18.53 | 6.48 | 8.01 | 1.61 | 4.31 | 0.3 | 18.47 | 5.27 | 5.78 | 0.23 |
| Llama 2 7B | 5.92 | 2.7 | 16.14 | 4.94 | 7.78 | 1.54 | 4.1 | 0.32 | 18.14 | 5.4 | 5.59 | 0.24 |
| Llama 3.1 8B | 24.98 | 12.21 | 26.04 | 15.15 | 43.02 | 21.62 | 7.12 | 1.71 | 19.59 | 8.86 | 14.61 | 4.31 |
| Gemma 2 9B | 8.27 | 4.8 | 22.35 | 11.86 | 5.12 | 1.43 | 3.89 | 0.38 | 25.29 | 10.56 | 5.62 | 0.25 |
| Phi 3.5 Mini | 6.31 | 3.22 | 13.64 | 5.22 | 5.4 | 1.19 | 3.3 | 0.44 | 9.67 | 3.38 | 4.14 | 0.21 |
Jailbreak Attacks
Jailbreak attacks exploit prompts, context, or model representations to bypass AI safeguards and elicit disallowed outputs.
See all 79 papers