SoK: Robustness in Large Language Models against Jailbreak Attacks

Large Language Models (LLMs) are increasingly used as interfaces to information, code, and real-world services, making prompt-level security failures a practical concern. Although jailbreak attacks, defenses, datasets, and automated judgers have advanced rapidly, evaluation remains fragmented across threat models, access assumptions, cost budgets, datasets, and success criteria. This makes reported attack success rates and defense gains hard to compare. This SoK systematizes LLM prompt security across concepts, data, tooling, and measurement. We propose linked taxonomies for jailbreak attacks, defenses, and model vulnerabilities, while separating technical mechanisms from attacker and defender capabilities. We also formalize threat, access, and cost assumptions as explicit evaluation metadata. To support reproducible evaluation, we release JailbreakDB, PromptSecurity-Eval, and PromptSecurity, a modular platform that represents each experiment as a tuple of model, attack, defense, dataset, and judger. Using matched evaluations across models, attacks, defenses, and judgers, we show that access regime, native harmful-query behavior, attack cost, defense backfire, taxonomy subcategory, and judger choice all materially affect security conclusions. Together, these artifacts support reproducible, cost-aware, and taxonomy-grounded evaluation of LLM prompt security. Leaderboard: https://datasec-lab.github.io/PromptSecurityLeaderboard/. Dataset: https://huggingface.co/datasets/youbin2014/JailbreakDB. GitHub: https://github.com/datasec-lab/PromptSecurity.

The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates. We propose JailMeter, an evidence-based evaluation framework designed to more faithfully measure jailbreak effectiveness. Inspired by the Information Bottleneck theory, JailMeter applies dual-feedback optimization to filter jailbreak noise from model responses while preserving content relevant to the original malicious question. This process produces concise evidence for a rigorous assessment under which an attack is validated only when the response captures the malicious intent and delivers a complete answer, thereby signaling a substantive bypass of model safety alignment. We evaluate JailMeter on JailMeter-Eva, a challenging benchmark containing 330 human-labeled, non-rejected jailbreak instances. JailMeter achieves an accuracy of 97.27%, substantially outperforming existing evaluation methods. To support large-scale evaluation, we further distill JailMeter into a small language model, JailMeter\textsubscript{SLM}, which maintains comparable reliability with significantly reduced computational costs. Code and dataset are available at https://github.com/Magi2B0y/JailMeter.

Despite the growing interest in jailbreaks as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed evaluation system designs have led to significant discrepancies in their effectiveness assessments. With a systematic measurement study based on 37 jailbreak studies since 2022, we find that existing evaluation systems lack case-specific criteria, resulting in misleading conclusions about their effectiveness and safety implications. In this paper, we introduce GuidedBench, a novel benchmark comprising a curated harmful question dataset and GuidedEval, an evaluation system integrated with detailed case-by-case evaluation guidelines. Experiments demonstrate that GuidedBench offers more accurate evaluations of jailbreak performance, enabling meaningful comparisons across methods. GuidedEval reduces inter-evaluator variance by at least 76.03%, ensuring reliable and reproducible evaluations. We reveal why existing jailbreak benchmarks fail to evaluate accurately and suggest better evaluation practices.