Large language models (LLMs) are increasingly deployed in user-facing systems, where jailbreak attacks can bypass safety controls and induce harmful and policy-violating outputs. Recent model-internal detectors use hidden representations, gradients, or internal signals to identify such prompts, but their practical security value depends critically on behavior at low false-positive operating points rather than on area under the receiver operating characteristic curve (AUROC) alone. We study two data dependencies that shape this behavior: calibration data, which determine the detection threshold, and reference data, which can influence detector scores. Across four model-internal detectors, four LLMs, and twelve jailbreak families, thresholds calibrated on curated benign prompts frequently fail to preserve their false-positive rate (FPR) budgets on real-user traffic. Recalibration improves false-positive control but reveals substantial family-level coverage gaps, including cases with high AUROC but low true-positive rate (TPR). Causal reference-set interventions show that changing reference composition can substantially alter family-level detection and that improving coverage of one attack family can reduce coverage of others. These findings show that calibration and reference data are part of a detector's security behavior. We therefore recommend evaluating model-internal jailbreak defenses using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points, with AUROC used as a complementary ranking metric rather than the primary security measure.
Figures & tables
Dataset
Type
Calibration
Evaluation
Dolly [ 12 ]
Curated instructions
200
Held-out
ToxicChat [ 38 ]
Real-user prompts
∼ 1K
500
WildChat [ 40 ]
Real-user prompts
1K
500
OR-Bench [ 14 ]
Difficult benign
–
Control
TABLE I: Benign datasets used for calibration and evaluation. Calibration and evaluation partitions are disjoint.
Llama-2
Llama-3
Mistral
Vicuna
Threshold from
Dolly
Dolly
Dolly
TC
Dolly
Dolly
Dolly
TC
Dolly
Dolly
Dolly
TC
Dolly
Dolly
Dolly
TC
Measured
FPR TC
FPR WC
Det.
Det.
FPR TC
FPR WC
Det.
Det.
FPR TC
FPR WC
Det.
Det.
FPR TC
FPR WC
Det.
Det.
MTK
0.236
0.286
0.965
0.570
0.116
0.136
0.933
0.742
0.272
0.368
0.968
0.608
0.252
0.284
0.909
0.554
Linear probe
0.198
0.232
0.954
0.705
0.124
0.174
0.880
0.683
0.166
0.256
0.765
0.506
0.164
0.152
0.765
0.403
GradSafe
0.078
0.096
0.775
0.697
0.110
0.186
0.856
0.646
0.304
0.286
0.754
0.350
0.260
0.258
0.744
0.410
HiddenDetect
0.078
0.100
0.620
0.488
0.068
0.052
0.131
0.085
0.026
0.016
0.037
0.094
0.098
0.120
0.772
0.529
TABLE II: Calibration transfer from curated to real-user benign prompts. Thresholds are calibrated to a nominal 5% false-positive budget using either Dolly or ToxicChat (TC). FPR TC and FPR WC denote the realized false-positive rates on held-out ToxicChat and WildChat, respectively; Det. is the mean detection rate across the twelve attack families.
Fig. 1: Calibration transfer and attack coverage under curated and real-user thresholds.
Llama-2
Llama-3
Mistral
Vicuna
Attack Family
n
AUROC
TPR@5%
AUROC
TPR@5%
AUROC
TPR@5%
AUROC
TPR@5%
SAA
500
0.942
0.114
0.918
0.002
0.981
1.000
0.850
0.026
Zulu
500
0.902
0.034
0.975
0.752
0.958
0.688
0.866
0.108
AutoDAN
500
0.969
0.936
0.984
0.956
0.925
0.108
0.961
0.744
nanoGCG
42/500/500/500
0.914
0.429
0.991
0.952
0.978
0.876
0.985
0.922
DrAttack
500
0.922
0.584
0.960
0.862
0.928
0.124
0.957
0.788
TABLE III: MTK per-family ranking and detection under real-user calibration. TPR is measured at the nominal 5% false-positive budget. Bold values indicate TPR <0.20 when AUROC ≥0.85 .
Fig. 2: Ranking quality and operating-point detection at the 5% real-user-calibrated threshold. Error bars show bootstrap uncertainty.
Fig. 3: MTK family-level detection versus realized FPR under real-user calibration. Gray curves denote the remaining attack families; the dashed line marks the nominal 5% operating point.
Fig. 4: MTK sensitivity across release seeds. Each point denotes one run and stars mark the configured seeds. The top panel relates aggregate AUROC to SAA detection at the 5% operating point; the bottom panel relates SAA AUROC to formatting artifacts sampled into the malicious reference bank.
Intervention
Llama-2
Llama-3
Mistral
Vicuna
None
.040/.610
.002/.785
.438/.472
.011/.563
Ordinary PKU
.000/.555
.002/.789
.357/.417
.009/.559
SAA-shaped
.706 /.564
1.000 /.727
.324/.095
1.000 /.492
SAA + shuffle
.670/.570
1.000 /.728
.339/.120
1.000 /.447
SAA + random
.376/.586
1.000 /.762
.000/.138
1.000 /.421
TABLE IV: Controlled reference interventions and family-level coverage at the recalibrated 5% operating point. Each entry reports SAA TPR / mean TPR over the remaining eleven attack families. Each entry is a mean of three banks; Ordinary PKU is the control.
Mean over 12 families
SAA
FPR
Model
Detector
AUROC
1%
5%
AUROC
1%
5%
1%
5%
Llama-2
MTK
0.935
0.125
0.570
0.942
0.000
0.114
0.014
0.054
GradSafe
0.937
0.220
0.697
0.957
0.000
0.812
0.010
0.044
HiddenDetect
0.893
0.079
0.488
0.765
0.000
0.000
0.012
0.044
Linear probe
0.950
0.363
0.705
0.882
0.000
0.002
0.012
0.040
Windowed PPL
0.574
0.171
0.252
1.000
1.000
1.000
0.006
0.060
TABLE V: Cross-detector performance at 1% and 5% false-positive budgets. Results report mean detection across the twelve attack families, SAA-specific performance, realized false-positive rates. AUROC is reported separately to distinguish ranking quality from operating-point detection. Not a ranking of detectors: GradSafe outside Llama-2 is our port, and HiddenDetect is our adaptation.
Published evaluations of prompt-injection and jailbreak detectors for Large Language Models often suffer from two systematic weaknesses: per-dataset threshold tuning and undisclosed operating points. We describe an evaluation harness that addresses both. The detector under evaluation is scored across 16 public benchmarks (12,111 samples) using 5-fold cross-validation. StratifiedKFold (by row) is the headline pass; a parallel StratifiedGroupKFold pass over a composite key (parent-prompt id plus MinHash + LSH near-duplicate clusters at Jaccard ≳0.8) runs alongside it as a leakage-premium diagnostic. A single global operating point is selected on the held-out folds (max F1 subject to FPR ≤1%) and applied uniformly to every dataset, so per-dataset results reflect one threshold rather than per-benchmark optimisation. Generalisation is examined through a battery of diagnostics (leave-one-dataset-out cross-validation, a random-label control, adversarial validation, permutation feature importance, length-bias correlation, classifier-head agreement, cross-source near-duplicate detection, threshold transferability, train-vs-OOF agreement, and a paraphrase-invariance probe), most with a quantitative pass threshold and the remainder with a stated failure mode. For every external comparison, the detector's threshold is re-tuned to the competitor's published false-positive rate so head-to-head values are evaluated at matched operating points.
Jailbreak prompts are a practical and evolving threat to large language models (LLMs), particularly in agentic systems that execute tools over untrusted content. Many attacks exploit long-context hiding, semantic camouflage, and lightweight obfuscations that can evade single-pass guardrails. We present RLM-JB, an end-to-end jailbreak detection framework built on Recursive Language Models (RLMs), in which a root model orchestrates a bounded analysis program that transforms the input, queries worker models over covered segments, and aggregates evidence into an auditable decision. RLM-JB treats detection as a procedure rather than a one-shot classification: it normalizes and de-obfuscates suspicious inputs, chunks text to reduce context dilution and guarantee coverage, performs parallel chunk screening, and composes cross-chunk signals to recover split-payload attacks. On AutoDAN-style adversarial inputs, RLM-JB achieves high detection effectiveness across three LLM backends (ASR/Recall 92.5-98.0%) while maintaining very high precision (98.99-100%) and low false positive rates (0.0-2.0%), highlighting a practical sensitivity-specificity trade-off as the screening backend changes.
Detecting jailbreak behaviour in large language models remains challenging, particularly when strongly aligned models produce harmful outputs only rarely. In this work, we present an empirical study of output based jailbreak detection under realistic conditions using the JailbreakBench Behaviors dataset and multiple generator models with varying alignment strengths. We evaluate both a lexical TF-IDF detector and a generation inconsistency based detector across different sampling budgets. Our results show that single output evaluation systematically underestimates jailbreak vulnerability, as increasing the number of sampled generations reveals additional harmful behaviour. The most significant improvements occur when moving from a single generation to moderate sampling, while larger sampling budgets yield diminishing returns. Cross generator experiments demonstrate that detection signals partially generalise across models, with stronger transfer observed within related model families. A category level analysis further reveals that lexical detectors capture a mixture of behavioural signals and topic specific cues, rather than purely harmful behaviour. Overall, our findings suggest that moderate multi sample auditing provides a more reliable and practical approach for estimating model vulnerability and improving jailbreak detection in large language models. Code will be released.
Hanrui Luo, Shreyank N Gowda
School of Computer Science, University of Nottingham, Nottingham, United Kingdom.