Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors
Organizations: Department of Cybersecurity University at Albany, SUNY Albany, NY, USA
Abstract
Large language models (LLMs) are increasingly deployed in user-facing systems, where jailbreak attacks can bypass safety controls and induce harmful and policy-violating outputs. Recent model-internal detectors use hidden representations, gradients, or internal signals to identify such prompts, but their practical security value depends critically on behavior at low false-positive operating points rather than on area under the receiver operating characteristic curve (AUROC) alone. We study two data dependencies that shape this behavior: calibration data, which determine the detection threshold, and reference data, which can influence detector scores. Across four model-internal detectors, four LLMs, and twelve jailbreak families, thresholds calibrated on curated benign prompts frequently fail to preserve their false-positive rate (FPR) budgets on real-user traffic. Recalibration improves false-positive control but reveals substantial family-level coverage gaps, including cases with high AUROC but low true-positive rate (TPR). Causal reference-set interventions show that changing reference composition can substantially alter family-level detection and that improving coverage of one attack family can reduce coverage of others. These findings show that calibration and reference data are part of a detector's security behavior. We therefore recommend evaluating model-internal jailbreak defenses using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points, with AUROC used as a complementary ranking metric rather than the primary security measure.
Figures & tables
| Dataset | Type | Calibration | Evaluation |
|---|---|---|---|
| Dolly [ 12 ] | Curated instructions | 200 | Held-out |
| ToxicChat [ 38 ] | Real-user prompts | 1K | 500 |
| WildChat [ 40 ] | Real-user prompts | 1K | 500 |
| OR-Bench [ 14 ] | Difficult benign | – | Control |
| Llama-2 | Llama-3 | Mistral | Vicuna | |||||||||||||
| Threshold from | Dolly | Dolly | Dolly | TC | Dolly | Dolly | Dolly | TC | Dolly | Dolly | Dolly | TC | Dolly | Dolly | Dolly | TC |
| Measured | FPR TC | FPR WC | Det. | Det. | FPR TC | FPR WC | Det. | Det. | FPR TC | FPR WC | Det. | Det. | FPR TC | FPR WC | Det. | Det. |
| MTK | 0.236 | 0.286 | 0.965 | 0.570 | 0.116 | 0.136 | 0.933 | 0.742 | 0.272 | 0.368 | 0.968 | 0.608 | 0.252 | 0.284 | 0.909 | 0.554 |
| Linear probe | 0.198 | 0.232 | 0.954 | 0.705 | 0.124 | 0.174 | 0.880 | 0.683 | 0.166 | 0.256 | 0.765 | 0.506 | 0.164 | 0.152 | 0.765 | 0.403 |
| GradSafe | 0.078 | 0.096 | 0.775 | 0.697 | 0.110 | 0.186 | 0.856 | 0.646 | 0.304 | 0.286 | 0.754 | 0.350 | 0.260 | 0.258 | 0.744 | 0.410 |
| HiddenDetect | 0.078 | 0.100 | 0.620 | 0.488 | 0.068 | 0.052 | 0.131 | 0.085 | 0.026 | 0.016 | 0.037 | 0.094 | 0.098 | 0.120 | 0.772 | 0.529 |
| Llama-2 | Llama-3 | Mistral | Vicuna | ||||||
| Attack Family | AUROC | TPR@5% | AUROC | TPR@5% | AUROC | TPR@5% | AUROC | TPR@5% | |
| SAA | 500 | 0.942 | 0.114 | 0.918 | 0.002 | 0.981 | 1.000 | 0.850 | 0.026 |
| Zulu | 500 | 0.902 | 0.034 | 0.975 | 0.752 | 0.958 | 0.688 | 0.866 | 0.108 |
| AutoDAN | 500 | 0.969 | 0.936 | 0.984 | 0.956 | 0.925 | 0.108 | 0.961 | 0.744 |
| nanoGCG | 42/500/500/500 | 0.914 | 0.429 | 0.991 | 0.952 | 0.978 | 0.876 | 0.985 | 0.922 |
| DrAttack | 500 | 0.922 | 0.584 | 0.960 | 0.862 | 0.928 | 0.124 | 0.957 | 0.788 |
| Intervention | Llama-2 | Llama-3 | Mistral | Vicuna |
|---|---|---|---|---|
| None | .040/.610 | .002/.785 | .438/.472 | .011/.563 |
| Ordinary PKU | .000/.555 | .002/.789 | .357/.417 | .009/.559 |
| SAA-shaped | .706 /.564 | 1.000 /.727 | .324/.095 | 1.000 /.492 |
| SAA + shuffle | .670/.570 | 1.000 /.728 | .339/.120 | 1.000 /.447 |
| SAA + random | .376/.586 | 1.000 /.762 | .000/.138 | 1.000 /.421 |
| Mean over 12 families | SAA | FPR | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Detector | AUROC | 1% | 5% | AUROC | 1% | 5% | 1% | 5% |
| Llama-2 | MTK | 0.935 | 0.125 | 0.570 | 0.942 | 0.000 | 0.114 | 0.014 | 0.054 |
| GradSafe | 0.937 | 0.220 | 0.697 | 0.957 | 0.000 | 0.812 | 0.010 | 0.044 | |
| HiddenDetect | 0.893 | 0.079 | 0.488 | 0.765 | 0.000 | 0.000 | 0.012 | 0.044 | |
| Linear probe | 0.950 | 0.363 | 0.705 | 0.882 | 0.000 | 0.002 | 0.012 | 0.040 | |
| Windowed PPL | 0.574 | 0.171 | 0.252 | 1.000 | 1.000 | 1.000 | 0.006 | 0.060 | |
Jailbreak Detection
Jailbreak detection identifies prompts and conversational strategies that try to bypass language models’ safety safeguards.
See all 45 papers