Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregation

Shayan Ali Hassan; Zaeem Mohtashim Khan; Tao Ni; Zafar Ayyub Qazi; Marco Canini

cs.LGFeb 8, 2026

Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregation

Authors: Shayan Ali Hassan, Zaeem Mohtashim Khan, Tao Ni, Zafar Ayyub Qazi, Marco Canini

Organizations: KAUST · LUMS & KAUST

Abstract

Large Language Models have demonstrated remarkable capabilities in natural language understanding. However, they remain susceptible to malicious prompts that induce unsafe or policy-violating behavior. Existing defenses in black-box moderation APIs offer limited transparency and adapt poorly to evolving threats, while white-box approaches using large LLM judges impose prohibitive computational costs and require expensive retraining for new attacks. We present BAGEL (Bootstrap AGgregated Ensemble Layer), a modular, lightweight, and incrementally updatable framework for malicious prompt detection. BAGEL employs a bootstrap aggregation and mixture-of-experts-inspired ensemble of finetuned models, each specialized on a different attack dataset. At inference, BAGEL aggregates predictions from a stochastically selected subset of ensemble members, optionally augmented by a lightweight learned router that predicts the most suitable member to include. When new attacks emerge, BAGEL updates incrementally by finetuning just a single small classifier (86M parameters) and adding it to the ensemble. We evaluate BAGEL on a held-out, source-disjoint set constructed from datasets never used for training or calibration, ensuring that performance reflects generalization to out-of-distribution attacks. In single-turn evaluation, BAGEL achieves an F1 score of 0.757 at a size equivalent to 430M parameters, improving to 0.772 when augmented with the router using fewer ensemble members (324M parameters). These results outperform popular black-box moderation APIs and most white-box baselines at lower costs. In multi-turn evaluation, BAGEL achieves an F1 score of 0.660, outperforming billion-parameter methods. Our results show ensembles of small finetuned classifiers can match or exceed the performance of billion-parameter guardrails while offering the adaptability and efficiency required for production systems.

Figures & tables

Figure 1: Examples of malicious prompt attack types. The malicious task has been redacted here to prevent direct inclusion of harmful content in this paper.

Figure 2: BAGEL overview, divided into two sections. The larger section details the ensemble selection strategy and probability aggregation. The smaller section details the dataset partitioning, finetuning and addition of a new promptcop to the ensemble during system updates.

Figure 4: $F_{1}$ before ( $k=9$ ) and after ( $k=10$ ) adding a promptcop trained on out-of-distribution data.

Method	ASR $\downarrow$	FPR $\downarrow$	F1 $\uparrow$	Bal. Acc. $\uparrow$	Parameters	Size Ratio	Fine-tunable?
BAGEL	0.342	0.307	0.757	0.675	430M for $n{=}5$	$1.0\times$	✓
BAGEL w/ DistilBERT	0.325	0.288	0.772	0.694	324M for $n{=}3$	$0.8\times$	✓
ToxicDetector	0.426	0.400	0.679	0.587	300M + 7B	$17.0\times$	✓
StruQ	0.065	0.732	0.880	0.602	7B	$16.3\times$	✓
ShieldGemma	0.431	0.068	0.717	0.751	2B	$4.7\times$	✓
OpenAIModerationAPI	0.528	0.089	0.631	0.692	Unknown	-	✗

Table 2: Results of evaluating BAGEL against other methodologies on the held-out test set. Size ratio is each method’s parameter count relative to BAGEL (430M,

n{=}5

	Strategy	F1 $\uparrow$	ASR $\downarrow$	FPR $\downarrow$
BAGEL ( $\bm{k}=9$ ) $\bm{n}=9$ $\bm{n}=5$ $\bm{n}=1$	Full-Conv.	0.636	0.232	0.593
	Chunk-Level	0.625	0.117	0.862
	Full-Conv.	0.660	0.106	0.748
	Chunk-Level	0.648	0.054	0.894
	Full-Conv.	0.704	0.155	0.508
	Chunk-Level	0.664	0.180	0.597

Table 3: Multi-turn attack detection performance (ToxicDetector with Chunk-Level is not evaluated due to training incompatibilities).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Value
Weights per promptcop	1.04 GB
Storage, full ensemble ( $k{=}9$ )	9.36 GB
Active memory, BAGEL ( $n{=}5$ )	5.20 GB
Active memory, BAGEL w/ DistilBERT ( $n{=}3$ )	3.12 GB + router
Forward pass per promptcop (batch size 64)	323.8 ms
Throughput per promptcop (batch size 64)	197.05 samples/s

Appendix

Table 4: Measured per-promptcop inference cost and resulting memory footprint of BAGEL .

Dataset	Number of Samples	Type of Malicious Prompts
	Number of Samples	Simple Harmful	Jailbreak	Injection
synapsecai/synthetic-prompt-injections ( SynapseCAI, 2024 )	252,956	$\circ$	$\circ$	$\bullet$
Malicious Prompt Detection Dataset (MPDD) ( Jebbar, 2025 )	39,234	$\bullet$	$\bullet$	$\bullet$
TrustAIRLab/in-the-wild-jailbreak-prompts ( Shen et al., 2024 )	13,700	$\circ$	$\bullet$	$\circ$
Harelix/Prompt-Injection-Mixed-Techniques-2024 ( Harelix, 2024 )	1,175	$\bullet$	$\circ$	$\circ$
jackhhao/jailbreak-classification ( Jack Hao, 2023 )	1,042	$\circ$	$\bullet$	$\circ$

Appendix

Table 5: Collection of the nine datasets we used for single-turn finetuning and calibration.

\bullet

indicates the dataset contains this specific type of malicious prompts, while

\circ

represents the lack of any such type.

Dataset	Malicious	Benign	Total
hlyn-labs/prompt-injection-judge-deberta-dataset ( hlyn, 2026 )	484	516	1,000
allenai/wildjailbreak ( Jiang et al., 2024 )	2,000	210	2,210
LLM Past Tense ( Andriushchenko and Flammarion, 2024 )	730	0	730
Necent/llm-jailbreak-prompt-injection-dataset ( Necent, 2026 )	629	371	1,000
TrustAIRLab/JailbreakQR ( Chu et al., 2025 )	400	0	400
Total	4,243	1,097	5,340

Appendix

Table 6: Collection of the five datasets used to construct the source-disjoint test set

\mathcal{T}

. All 210 benign WildJailbreak prompts come from its adversarial_benign split.

Figure 5: Pairwise Jensen-Shannon divergence between the nine training datasets and the five OOD test datasets. Prompts are embedded with all-mpnet-base-v2 , the shared embedding space is partitioned into 50 clusters via $k$ -means, and each dataset’s distribution is represented as its normalized histogram of prompts per cluster. Darker cells indicate greater distributional similarity between a pair of datasets.

LLM Safety Monitoring

See all 54 papers

Mar 26, 2026

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

Prompt attacks, including jailbreaks and prompt injections, pose a critical security risk to Large Language Model (LLM) systems. In production, guardrails must mitigate these attacks under strict low-latency constraints, resulting in a deployment gap in which lightweight classifiers and rule-based systems struggle to generalize under distribution shift, while high-capacity LLM-based judges remain too slow or costly for live enforcement. In this work, we examine whether lightweight, general-purpose LLMs can reliably serve as security judges under real-world production constraints. Through careful prompt and output design, lightweight LLMs are guided through a structured reasoning process involving explicit intent decomposition, safety-signal verification, harm assessment, and self-reflection. We evaluate our method on a curated dataset combining benign queries from real-world chatbots with adversarial prompts generated via automated red teaming (ART), covering diverse and evolving patterns. Our results show that general-purpose LLMs, such as gemini-2.0-flash-lite-001, can serve as effective low-latency judges for live guardrails. This configuration is currently deployed in production as a centralized guardrail service for public service chatbots in Singapore. We additionally evaluate a Mixture-of-Models (MoM) setting to assess whether aggregating multiple LLM judges improves prompt-attack detection performance relative to single-model judges, with only modest gains observed.

Hieu Xuan Le, Benjamin Goh, Quy Anh Tang

GovTech, Singapore

Prompt Injection Detection LLM Guardrails LLM Security LLM-as-a-Judge

Ask PDF

Jun 1, 2026

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

Published evaluations of prompt-injection and jailbreak detectors for Large Language Models often suffer from two systematic weaknesses: per-dataset threshold tuning and undisclosed operating points. We describe an evaluation harness that addresses both. The detector under evaluation is scored across 16 public benchmarks (12,111 samples) using 5-fold cross-validation. StratifiedKFold (by row) is the headline pass; a parallel StratifiedGroupKFold pass over a composite key (parent-prompt id plus MinHash + LSH near-duplicate clusters at Jaccard

\gtrsim 0.8

) runs alongside it as a leakage-premium diagnostic. A single global operating point is selected on the held-out folds (max F1 subject to FPR

\leq 1\%

) and applied uniformly to every dataset, so per-dataset results reflect one threshold rather than per-benchmark optimisation. Generalisation is examined through a battery of diagnostics (leave-one-dataset-out cross-validation, a random-label control, adversarial validation, permutation feature importance, length-bias correlation, classifier-head agreement, cross-source near-duplicate detection, threshold transferability, train-vs-OOF agreement, and a paraphrase-invariance probe), most with a quantitative pass threshold and the remainder with a stated failure mode. For every external comparison, the detector's threshold is re-tuned to the competitor's published false-positive rate so head-to-head values are evaluated at matched operating points.

Ryle Goehausen, Marcus Sousa

LLM Safety Benchmarks Language Model Safety Evaluation LLM Security Jailbreak Detection Prompt Injection Attacks on LLMs Prompt Injection Detection

Ask PDF

Jul 29, 2026

RoguePrompt: Dual-Layer Encoding for Self-Reconstruction to Circumvent LLM Moderation

Large language models (LLMs) are becoming increasingly integrated into mainstream development platforms and daily technological workflows, typically behind moderation and safety controls. Despite these controls, preventing prompt-based policy evasion remains challenging, and adversaries continue to "jailbreak" LLMs by crafting prompts that circumvent implemented safety mechanisms. Prior work has established cipher-mediated interaction, code-embedded decryption, prompt decomposition and reconstruction, and layered custom encryption as viable attack primitives. However, reported evaluations generally collapse visible acceptance, successful recovery of the concealed request, and subsequent execution into an aggregate attack-success outcome. This leaves limited evidence about where multistage prompt-transformation attacks fail within an observable black-box interaction. This paper introduces RoguePrompt, a jailbreak pipeline that partitions a forbidden prompt and applies two nested encodings, Vigenere followed by ROT13, along with natural-language reconstruction instructions. RoguePrompt was developed and evaluated under a black-box threat model, with only API or user-interface access to the hosted models, and was tested on 313 real-world, hard-rejected prompts. Success was measured in terms of moderation bypass, instruction reconstruction, and execution when the relevant stage exceeded its automated criterion. RoguePrompt achieved average rates of 93.93% for filter bypass, 79.02% for reconstruction, and 70.18% for execution. These results demonstrate the effectiveness of layered prompt encoding while providing stage-level evidence of where multistage jailbreaks fail during moderation bypass, instruction reconstruction, and execution.

Benyamin Tafreshian, Prathamesh Dhake

Department of Computer Science · Boston University · Boston, MA, USA

Adversarial Attacks Adversarial Attacks on LLMs Safety Filtering LLM Jailbreak Attacks LLM Safety Evaluation Prompt Injection Attacks on LLMs

Ask PDF

Abstract

LLM Safety Monitoring

Explore similar work

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

RoguePrompt: Dual-Layer Encoding for Self-Reconstruction to Circumvent LLM Moderation