SecureBreak -- A dataset towards safe and secure models

Marco Arazzi; Vignesh Kumar Kembu; Antonino Nocera

cs.CRMar 23, 2026

SecureBreak -- A dataset towards safe and secure models

Authors: Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera

Organizations: Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy

Abstract

Large language models are becoming pervasive core components in many real-world applications. As a consequence, security alignment represents a critical requirement for their safe deployment. Although previous related works focused primarily on model architectures and alignment methodologies, these approaches alone cannot ensure the complete elimination of harmful generations. This concern is reinforced by the growing body of scientific literature showing that attacks, such as jailbreaking and prompt injection, can bypass existing security alignment mechanisms. As a consequence, additional security strategies are needed both to provide qualitative feedback on the robustness of the obtained security alignment at the training stage, and to create an ``ultimate'' defense layer to block unsafe outputs possibly produced by deployed models. To provide a contribution in this scenario, this paper introduces SecureBreak, a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs caused by residual weaknesses in security alignment. The dataset is highly reliable due to careful manual annotation, where labels are assigned conservatively to ensure safety. It performs well in detecting unsafe content across multiple risk categories. Tests with pre-trained LLMs show improved results after fine-tuning on SecureBreak. Overall, the dataset is useful both for post-generation safety filtering and for guiding further model alignment and security improvements.

Figures & tables

Dataset	Primary Target	Context Scope	Annotation Method	Task Type	Primary Utility
AdvBench [ 21 ]	Prompt	Adversarial (Jailbreak)	N/A (List)	Attack Success	Red Teaming (Attack)
HH-RLHF [ 10 ]	Prompt & Response	General Safety	Human	Preference (A vs B)	Alignment (RLHF)
BeaverTails [ 11 ]	Prompt & Response	General Safety	LLM + Human	Classification	Safety Alignment
JailbreakBench [ 3 ]	Prompt	Adversarial (Jailbreak)	Automated (LLM)	Attack Success	Benchmarking Attacks
Do Not Answer [ 12 ]	Instruction	Risk Guidelines	Human	Evaluation	Instruction Following
SecureBreak (Ours)	Response	Adversarial (Jailbreak)	Human (Expert)	Binary Classification	Judge Alignment (Filtering)

TABLE I : Comparison of SecureBreak with existing prominent safety and jailbreak datasets.

No.	Features/Variables	Type	Description
1	Question	String	Harmful question which used as a input to the LLMs
2	Behavior	String	Which behavior does the question belong, individual question wise.
3	Category	String	Broad spectrum of behaviors clubbed together.
4	Question_Source	String	From were the question was gathered.
5	Response	String	LLMs response to the question.
6	LLMs	String	From which LLM the response is generated.

TABLE II : Description of the 9 columns in the SecureBreak dataset, including 1 float and 8 string feature columns.

Fig. 2 : Data distribution according to the annotated class i.e., 0 - Safe and 1 - Unsafe

Fig. 3 : Class-wise data distribution w.r.t top 10 behaviors

Fig. 4 : Class-wise data distribution w.r.t categories

Fig. 5 : Class-wise percentage distribution w.r.t LLMs

Model	Base	Fine-tuned QLoRA	SecureBreak Model (Seq2Seq Fine-tuned)
Mistral - 7B - v0.3	62.84	83.24	90.14
Llama 3.1 - 8B	61.89	81.22
Selene-1-Mini-Llama-3.1-8B	57.43	76.08

TABLE III : Transposed table with SecureBreak model kept separate as in the original layout.

Panel of Figure 9 — Fig. 6 : Base vs Fine-tuned model category wise accuracy comparison for the classification task and Seq2Seq Fine-tuned model accuracy category wise for classification task.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 438 papers

MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
May 3, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +4LLM Safety Benchmarks Adversarial Prompt Generation
Simon Fraser University · Microsoft Research
DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning
May 29, 2026Junbo Zhang, Qianli Zhou, Xinyang Deng +3LLM Fine-Tuning Instruction-Tuning Data Selection
Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment
May 1, 2026Tung-Ling Li, Hongliang Liu, Yuhao WuLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Palo Alto Networks

May 3, 2026J/K move · Enter open · S save

MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety

Jialin Song, Xiaodong Liu, Weiwei Yang, Wuyang Chen+3

Simon Fraser University · Microsoft Research

LLM Safety Benchmarks Adversarial Prompt Generation Language Model Safety Evaluation Adversarial Scenario Generation Jailbreak Attacks LLM Jailbreak Attacks+1

Open paper AskPDF arXiv

We present MultiBreak, a scalable and diverse multi-turn jailbreak benchmark to evaluate large language model (LLM) safety. Multi-turn jailbreaks mimic natural conversational settings, making them easier to bypass safety-aligned LLM than single-turn jailbreaks. Existing multi-turn benchmarks are limited in size or rely heavily on templates, which restrict their diversity. To address this gap, we unify a wide range of harmful jailbreak intents, and introduce an active learning pipeline for expanding high-quality multi-turn adversarial prompts, where a generator is iteratively fine-tuned to produce stronger attack candidates, guided by uncertainty-based refinement. Our MultiBreak includes 10,389 multi-turn adversarial prompts, spans 2,665 distinct harmful intents, and covers the most diverse set of topics to date. Empirical evaluation shows that our benchmark achieves up to a 54.0 and 34.6 higher attack success rate (ASR)} than the second-best dataset on DeepSeek-R1-7B and GPT-4.1-mini, respectively. More importantly, safety evaluations suggest that diverse attack categories uncover fine-grained LLM vulnerabilities}, and categories that appear benign under single-turn can exhibit substantially higher adversarial effectiveness in multi-turn scenarios. These findings highlight persistent vulnerabilities of LLMs under realistic adversarial settings and establish MultiBreak as a scalable resource for advancing LLM safety.

Abstract

LLM Safety

Explore similar work

MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety

DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning

Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment