cs.CRMar 24, 2026

Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs

Authors: Wenyu Chen, Xiangtao Meng, Chuanchao Zang, Li Wang, Xinyu Gao, Jianing Wang, Peng Zhan, Zheng Li, 

Organizations: Shandong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models(LLMs) are widely deployed, yet are vulnerable to jailbreak prompts that elicit policy-violating outputs. Although prior studies have uncovered these risks, they typically treat all tokens as equally important during prompt mutation, overlooking the varying contributions of individual tokens to triggering model refusals. Consequently, these attacks introduce substantial redundant searching under query-constrained scenarios, reducing attack efficiency and hindering comprehensive vulnerability assessment. In this work, we conduct a token-level analysis of refusal behavior and observe that token contributions are highly skewed rather than uniform. Moreover, we find strong cross-model consistency in refusal tendencies, enabling the use of a surrogate model to estimate token-level contributions to the target model's refusals. Motivated by these findings, we propose TriageFuzz, a token-aware jailbreak fuzzing framework that adapts the fuzz testing approach with a series of customized designs. TriageFuzz leverages a surrogate model to estimate the contribution of individual tokens to refusal behaviors, enabling the identification of sensitive regions within the prompt. Furthermore, it incorporates a refusal-guided evolutionary strategy that adaptively weights candidate prompts with a lightweight scorer to steer the evolution toward bypassing safety constraints. Extensive experiments on six open-source LLMs and three commercial APIs demonstrate that TriageFuzz achieves comparable attack success rates (ASR) with significantly reduced query costs. Notably, it attains a 90% ASR with over 70% fewer queries compared to baselines. Even under an extremely restrictive budget of 25 queries, TriageFuzz outperforms existing methods, improving ASR by 20-40%.

Figures & tables

Figure 1: Comparison between traditional uniform mutation and TriageFuzz\mathsf{TriageFuzz} .
Figure 2: Layer-wise comparison of cross-model Spearman correlations for refusal tendency across the first, reference ( l⋆l^{\star} ), and final layers.
Figure 3: Cross-model cosine similarity of the representation-wise refusal semantic features.
Figure 4: Schematic overview of TriageFuzz\mathsf{TriageFuzz} . The process iteratively refines prompts through token-level importance estimation, semantics-aware region mutation, and refusal-guided evolution.
Figure 5: Visualization of token-level refusal importance ( stokens_{\text{token}} ). The distribution is highly skewed.
Figure 6: The prompt template used to extract semantically coherent trigger spans from score-annotated sequences.
Figure 7: ASR–budget curves of TriageFuzz\mathsf{TriageFuzz} and representative black-box jailbreak baselines across six target models.
Target ModelBud- getGCGAuto DANPAIRGPT FuzzTAPTriageFuzz\mathsf{TriageFuzz}
Gemma -7B100.0%29.0%46.0%34.0%45.5%70.5%
150.5%44.5%58.0%36.0%59.0%83.0%
201.0%46.5%65.5%36.5%63.5%93.0%
251.0%47.5%74.0%36.5%73.0%96.0%
Gemma 2-9B100.0%30.5%31.0%27.5%39.0%59.0%
150.0%43.5%41.0%35.5%47.5%72.0%
Table 1: Per-model early-stage ASR under low query budgets.
Figure 8: Ablation study analyzing the impact of mutation strategies. We compare the Token-Aware approach ( TriageFuzz\mathsf{TriageFuzz} ) against a Uniform mutation baseline across varying query budgets.
Figure 9: Robustness of TriageFuzz\mathsf{TriageFuzz} to the selection of surrogate and attack models.
BudgetGPT-3.5-TurboGPT-4oClaude-3.5-Sonnet
PAIRGPTFuzzTAPOursPAIRGPTFuzzTAPOursPAIRGPTFuzzTAPOurs
ASR@1049.5%32.5%40.5%68.0%38.5%3.0%32.0%50.0%6.5%0.0%3.5%47.5%
ASR@1560.5%42.0%48.5%81.0%48.5%3.0%34.5%66.0%13.0%0.0%9.5%60.5%
ASR@2068.0%45.5%54.5%88.5%55.5%3.0%42.5%74.5%20.5%1.0%17.0%72.0%
ASR@2574.0%52.0%58.0%94.0%61.5%5.0%52.0%84.0%28.0%3.0%24.5%80.5%
Table 2: ASR on commercial closed-source APIs under low query budgets.
Target ModelSetting@10@15@20@25Avg
Gemma -7BBefore70.5%83.0%93.0%96.0%85.6%
After61.0%72.0%79.0%81.0%73.3%
Gemma 2-9BBefore59.0%72.0%82.5%93.5%76.8%
After44.5%53.0%61.0%71.0%57.4%
LLaMA 3-8BBefore42.5%62.5%80.0%88.0%68.3%
After31.5%49.0%63.0%70.5%53.5%
Table 3: Effect of SmoothLLM on ASR under low budgets.
Figure 10: Performance comparison under hybrid defenses.
Figure 11: Reference layer of the model.
Figure 12: System Prompt Template for Context-Adaptive Mutation.
MethodASR@10ASR@15ASR@20ASR@25
GCG0.08%0.17%0.50%0.92%
AUTODAN25.25%35.08%38.08%40.58%
PAIR36.83%48.00%55.08%61.75%
GPTFuzz23.34%28.34%32.42%34.58%
TAP42.67%52.67%57.17%63.58%
TriageFuzz\mathsf{TriageFuzz}63.75%76.50%86.08%92.17%
Table 4: Aggregated ASR comparison across all target models.
Target ModelLLaMA3.1-8bQwen2-7bMistral-7b
 @10@15@20@25@10@15@20@25@10@15@20@25
Gemma-7B70.5%83.0%93.0%96.0%64.0%79.0%88.5%94.5%66.0%79.5%89.5%93.0%
Gemma2-9B59.0%72.0%82.5%93.5%57.0%72.0%83.0%89.5%58.0%73.0%81.0%90.5%
LLaMA3-8B42.5%62.5%80.0%88.0%48.0%63.0%79.0%85.5%52.5%70.0%78.5%86.5%
LLaMA3.2-3B48.5%65.0%78.0%88.0%46.5%63.0%75.5%85.5%44.5%67.0%76.5%86.5%
Qwen2.5-7B79.0%86.5%91.5%94.5%81.5%89.5%92.0%95.5%78.0%84.0%88.5%93.5%
Table 5: Surrogate Model Ablation: Early-Stage ASR under Low Query Budgets.
Target ModelVicuna-13bLLaMA3-8bGemma3-12b
 @10@15@20@25@10@15@20@25@10@15@20@25
Gemma-7B70.5%83.0%93.0%96.0%68.0%75.5%88.5%92.0%87.5%91.0%95.0%96.5%
Gemma2-9B59.0%72.0%82.5%93.5%59.0%75.5%83.5%89.5%82.5%90.0%92.0%94.0%
LLaMA3-8B42.5%62.5%80.0%88.0%59.0%71.5%79.0%86.5%77.0%85.0%87.5%90.5%
LLaMA3.2-3B48.5%65.0%78.0%88.0%47.5%63.0%73.5%83.5%63.5%76.0%83.5%89.5%
Qwen2.5-7B79.0%86.5%91.5%94.5%61.5%75.0%84.5%91.0%93.0%93.5%94.5%96.5%
Table 6: Attack Model Ablation: Early-Stage ASR under Low Query Budgets.
Target ModelPPL FilterLLaMA GuardSmoothLLM
@10@15@20@25@10@15@20@25@10@15@20@25
Gemma-7B70.5%83.0%92.5%95.5%45.5%57.0%64.5%66.0%61.0%72.0%79.0%81.0%
Gemma2-9B56.5%68.5%79.0%90.0%36.0%46.0%52.5%60.5%44.5%53.0%61.0%71.0%
LLaMA3-8B42.5%62.5%80.0%88.0%24.0%34.5%46.5%51.5%31.5%49.0%63.0%70.5%
LLaMA3.2-3B47.0%63.5%75.0%84.5%32.0%44.5%52.5%59.5%42.5%57.5%67.5%77.0%
Qwen2.5-7B78.5%85.5%90.0%93.0%44.5%47.5%51.0%52.5%55.0%61.5%65.0%68.0%
Table 7: Comparison of defense mechanisms under limited query budgets.
More in this topic

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 322 papers

Explore similar work

CardsList
Feb 24, 2026

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.

Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2

NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2

Prompt Injection Attacks on LLMsLanguage Model Safety EvaluationLLM Jailbreak AttacksPrompt Injection Defense
AskPDF
Mar 3, 2026

TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models

Large language models (LLMs) have achieved remarkable success across diverse applications but remain vulnerable to jailbreak attacks, where attackers craft prompts that bypass safety alignment and elicit unsafe responses. Among existing approaches, optimization-based attacks have shown strong effectiveness, yet current methods often suffer from frequent refusals, pseudo-harmful outputs, and inefficient token-level updates. In this work, we propose TAO-Attack, a new optimization-based jailbreak method. TAO-Attack employs a two-stage loss function: the first stage suppresses refusals to ensure the model continues harmful prefixes, while the second stage penalizes pseudo-harmful outputs and encourages the model toward more harmful completions. In addition, we design a direction-priority token optimization (DPTO) strategy that improves efficiency by aligning candidates with the gradient direction before considering update magnitude. Extensive experiments on multiple LLMs demonstrate that TAO-Attack consistently outperforms state-of-the-art methods, achieving higher attack success rates and even reaching 100% in certain scenarios.

Zhi Xu, Jiaqi Li, Xiaotong Zhang +2

Dalian University of Technology, Dalian, China

Adversarial Attacks on LLMsJailbreak AttacksLLM Jailbreak Attacks
AskPDF
Apr 14, 2026

TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

Large Language Models (LLMs) are increasingly deployed across diverse domains, yet their vulnerability to jailbreak attacks, where adversarial inputs bypass safety mechanisms to elicit harmful outputs, poses significant security risks. While prior work has primarily focused on prompt injection attacks, these approaches often require resource-intensive prompt engineering and overlook other critical components, such as chat templates. This paper introduces TEMPLATEFUZZ, a fine-grained fuzzing framework that systematically exposes vulnerabilities in chat templates, a critical yet underexplored attack surface in LLMs. Specifically, TEMPLATEFUZZ (1) designs a series of element-level mutation rules to generate diverse chat template variants, (2) proposes a heuristic search strategy to guide the chat template generation toward the direction of amplifying the attack success rate (ASR) while preserving model accuracy, and (3) integrates an active learning-based strategy to derive a lightweight rule-based oracle for accurate and efficient jailbreak evaluation. Evaluated on twelve open-source LLMs across multiple attack scenarios, TEMPLATEFUZZ achieves an average ASR of 98.2% with only 1.1% accuracy degradation, outperforming state-of-the-art methods by 9.1%-47.9% in ASR and 8.4% in accuracy degradation. Moreover, even on five industry-leading commercial LLMs where chat templates cannot be specified, TEMPLATEFUZZ attains a 90% average ASR via chat template-based prompt injection attacks.

Qingchao Shen, Zibo Xiao, Lili Huang +3

School of Computer Software, Tianjin University, Tianjin, China · Monash University, Monash, Australia

Adversarial Attacks on LLMsLLM SecurityLLM Jailbreak AttacksAdversarial Prompt GenerationPrompt Injection Attacks on LLMs
AskPDF