TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

Qingchao Shen; Zibo Xiao; Lili Huang; Enwei Hu; Yongqiang Tian; Junjie Chen

cs.CRApr 14, 2026

TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

Authors: Qingchao Shen, Zibo Xiao, Lili Huang, Enwei Hu, Yongqiang Tian, Junjie Chen

Organizations: School of Computer Software, Tianjin University, Tianjin, China · Monash University, Monash, Australia

Abstract

Large Language Models (LLMs) are increasingly deployed across diverse domains, yet their vulnerability to jailbreak attacks, where adversarial inputs bypass safety mechanisms to elicit harmful outputs, poses significant security risks. While prior work has primarily focused on prompt injection attacks, these approaches often require resource-intensive prompt engineering and overlook other critical components, such as chat templates. This paper introduces TEMPLATEFUZZ, a fine-grained fuzzing framework that systematically exposes vulnerabilities in chat templates, a critical yet underexplored attack surface in LLMs. Specifically, TEMPLATEFUZZ (1) designs a series of element-level mutation rules to generate diverse chat template variants, (2) proposes a heuristic search strategy to guide the chat template generation toward the direction of amplifying the attack success rate (ASR) while preserving model accuracy, and (3) integrates an active learning-based strategy to derive a lightweight rule-based oracle for accurate and efficient jailbreak evaluation. Evaluated on twelve open-source LLMs across multiple attack scenarios, TEMPLATEFUZZ achieves an average ASR of 98.2% with only 1.1% accuracy degradation, outperforming state-of-the-art methods by 9.1%-47.9% in ASR and 8.4% in accuracy degradation. Moreover, even on five industry-leading commercial LLMs where chat templates cannot be specified, TEMPLATEFUZZ attains a 90% average ASR via chat template-based prompt injection attacks.

Figures & tables

Figure 1. This figure shows the responses of Meta-Llama-3-8B-Instruct to a harmful question under different chat templates. (1) With the default chat template, the model refuses to respond (response 1); without a template, it produces meaningless repetitions (response 2); under a prompt-based attack with the default template or with the TemplateFuzz -generated template, it generates harmful outputs (responses 3 and 4). (2) While both the prompt-based and TemplateFuzz attacks succeed, the former consumes far more tokens ( $\approx$ 200) than the latter ( $\approx$ 25).

Figure 2. This figure shows an example of a chat template with prompts in Qwen2.5-7B-Instruct and highlights its five key structural elements: System Message, User/Assistant Message, Role Marker, Delimiter, and Generation Hint.

Figure 3. Overview of TemplateFuzz . TemplateFuzz enables fuzz-based jailbreak attacks through three core components. The Chat Template Mutation module defines a set of fine-grained mutation rules for diverse chat template generation. The Heuristic Search Strategy module leverages the sampling learning and dynamic feedback strategies to efficiently guide chat template generation. The Learning-based Judge module adopts an active learning strategy to build a lightweight rule-based oracle for accurate and efficient jailbreak evaluation.

Table 1. This table presents an example for each chat template-based mutation rule, with red-highlighted text denoting the mutated content.

Figure 4. This figure presents the overall performance comparison of TemplateFuzz with baselines under the multi-question jailbreak scenario. Attack effectiveness is measured using Top-1 and Top-5 ASR, highlighting the universality and effectiveness of TemplateFuzz in exploiting the chat template of LLMs.

Method	Budget=50		Budget=100
Method	JQ	AQJ	JQ	AQJ
GPTFuzzer	8/50	9.38	15/50	38.70
TurboFuzzLLM	20/50	25.05	33/50	46.12
TemplateFuzz	45/50	9.31	47/50	12.89

Table 2. This table compares three fuzzing methods under the single-question jailbreak scenario, using 50 harmful questions from AdvBench, and reports the number of jailbroken questions (JQ) and the average queries to jailbreak (AQJ) under query budgets of 50 and 100.

Model	TemplateFuzz		ChatBug
	Top-1 Template		Mismatch- $\emptyset$		Mismatch-V		Mismatch-C
	ASR	ACC	ASR	ACC	ASR	ACC	ASR	ACC
Gemma3-4B	99.04%	1.01% $\downarrow$	21.54%	6.99% $\downarrow$	19.42%	9.10% $\downarrow$	27.12%	6.53% $\downarrow$
DeepSeek-7B	98.84%	1.41% $\downarrow$	73.08%	6.62% $\downarrow$	12.88%	3.72% $\downarrow$	26.73%	13.04% $\uparrow$
Llama2-7B	95.58%	0.21% $\uparrow$	28.85%	1.18% $\downarrow$	0.96%	1.71% $\downarrow$	4.04%	17.20% $\downarrow$
Qwen2.5-7B	90.96%	0.04% $\downarrow$	5.78%	22.65% $\downarrow$	1.35%	27.92% $\downarrow$	7.88%	3.28% $\downarrow$

Table 3. This table presents the ASR and model accuracy (ACC) of TemplateFuzz and ChatBug. ChatBug includes three attack variants: Mismatch-

\emptyset

deletes the default chat template, Mismatch-C replaces the default chat template with ChatML, and Mismatch-V substitutes the default chat template with the simplified format used by Vicuna.

Mutation Rules	ASR(ACC)
Mutation Rules	Top-1	Top-5
Direct Attack (without mutation)	0.00% (0.00%)	0.00%(0.00%)
System Message (M1)	28.84% (1.46%↓)	51.92% (1.72%↓)
User/Assistant Message (M2)	60.38% (1.83%↓)	78.65% (1.99%↓)
Role Marker (M3)	27.30% (1.33%↓)	41.92% (2.27%↓)
Delimiter (M4)	55.19% (0.62%↓)	75.19% (1.55%↓)

Table 4. This table presents the effectiveness of each mutation rule in TemplateFuzz (100 rounds).

Variants	Top-1 ASR (ACC)	Top-5 ASR (ACC)
TemplateFuzz no_sample	87.88% (1.04%↓)	100% (1.60%↓)
TemplateFuzz random	74.04% (3.44%↓)	86.35% (2.01%↓)
TemplateFuzz genetic	83.27% (4.01%↓)	98.08% (2.53%↓)
TemplateFuzz	95.58% (0.21%↑)	100% (0.99%↓)

Table 5. This table illustrates the ASR and model accuracy (ACC) performance of TemplateFuzz and its three alternative variants in 100 rounds fuzzing.

Strategy	Accuracy	TPR	FPR	Time
Rule Match origin	83.46%	78.08%	11.15%	$<$ 1s
Judge Model	89.42%	97.50%	2.53%	1,425s
Rule Match enhance	88.27%	92.69%	8.02%	$<$ 1s

Table 6. This table compares three jailbreak judge strategies on accuracy, TPR (true positive rate), FPR (false positive rate), and the time required to process 520 responses. An ideal strategy achieves high accuracy and TPR while maintaining low FPR and minimal time cost.

Model	Top-1 ASR	Top-5 ASR
GPT-4	55%	80%
Gemini-2.5-Flash	69%	89%
Qwen-Plus	40%	100%
DeepSeek-Chat	51%	81%
DeepSeek-Reasoner	100%	100%

Table 7. This table illustrates Top-1 and Top-5 ASR of TemplateFuzz on five commercial LLMs via chat template-based prompt injection attack.

Adversarial Attacks on LLMs

Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.

See all 355 papers

Automated jailbreak attack targeting multiple defense strategies
Jun 15, 2026Qi Wang, Chengcheng Wan, Weijia He +4Adversarial Prompt Generation LLM Jailbreak Attacks
East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England +2
Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs
Mar 24, 2026Wenyu Chen, Xiangtao Meng, Chuanchao Zang +6LLM Security LLM Jailbreak Attacks
Shandong University
Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
Feb 24, 2026Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2Prompt Injection Attacks on LLMs Language Model Safety Evaluation
NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2

Jun 15, 2026J/K move · Enter open · S save

Automated jailbreak attack targeting multiple defense strategies

Qi Wang, Chengcheng Wan, Weijia He, Yanqing Li+3

East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England+2

Adversarial Prompt Generation LLM Jailbreak Attacks LLM Safety Evaluation Prompt Injection Attacks on LLMs Black-Box Adversarial Attacks Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks. In this paper, we present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective to systematically construct effective black-box attack prompts. Unlike prior approaches that rely on static templates or iterative model-specific tuning, UNIATTACK extracts minimal but high-impact attack features from diverse existing attacks, optimizes them via a specialized attacker LLM, and composes them into flexible templates through automated refinement process. This feature-centric construction enables one-shot attacks that generalize across multiple models and safety categories, providing a practical tool for assessing LLM robustness. Our evaluation results shows that compared to the baselines, UNIATTACK achieves an average attack success rate (ASR) improvement of 64.63%-248.82% on models deployed with multi-layered defense mechanisms and it only takes 0.03%-4.96% cost of the baselines. UNIATTACK artifact is available at https://anonymous.4open.science/r/UniAttack-Artifact-30F1.

Abstract

Adversarial Attacks on LLMs

Explore similar work

Automated jailbreak attack targeting multiple defense strategies

Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks