System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks. Adversarial queries can successfully elicit these hidden instructions, posing significant security and privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt. Our core contribution is the formalization of prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm
Figures & tables
Dataset
Attack
Defense
GPT-5-mini
GPT-4.1-mini
GPT-4o-mini
AM Avg
JM Avg
AM Avg
JM Avg
AM Avg
JM Avg
Synthetic System Prompts
Liang
No Defense
0%
54%
26%
78%
0%
32%
Fake
0%
54%
24%
62%
1%
40%
Direct
0%
40%
3%
47%
0%
16%
PSM
0%
13%
0%
4%
0%
6%
Zhang
No Defense
1%
42%
26%
34%
3%
27%
Table 1: Attack success rates (%) on the UNNATURAL and Synthetic System Prompts corpora, averaged over attacks (AM = Approximate Match, JM = Judge Match). Lower is better; bold denotes the lowest value within each dataset × attack block.
Dataset
Attack
Defense
GPT-5-mini
GPT-4.1-mini
GPT-4o-mini
JM Avg
JM Avg
JM Avg
Synthetic System Prompts
Raccoon
No Defense
42%
59%
27%
Filter
18%
3%
5%
PSM
8%
5%
4%
Raccoon-Language
No Defense
35%
59%
17%
Filter
32%
48%
11%
Table 2: Judge-Match (JM) attack success rates (%) for Raccoon and Raccoon-Language suites on the UNNATURAL and Synthetic System Prompts corpora. Lower is better; bold denotes the lowest value within each dataset × attack block.
Dataset
GPT-5-mini
GPT-4.1-mini
GPT-4o-mini
Synthetic System Prompts
101.88%
100.89%
99.73%
UNNATURAL
101.27%
114.76%
100.73%
Table 3: Average utility preservation (%) for each model across system prompt corpora (Synthetic System Prompts and UNNATURAL), computed using the same definition as in Section Utility Objective .
Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.
Tushar Nayan, Pratik Poudel, Qian Lou +3
Florida International University · University of Central Florida · University of Texas at Arlington
LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.
We introduce a novel framework that transforms the resource-intensive (adversarial) prompt optimization problem into an efficient, amortized inference task. Our core insight is that pretrained, non-autoregressive generative LLMs, such as Diffusion LLMs, which model the joint distribution over prompt-response pairs, can serve as powerful surrogates for prompt search. This approach enables direct conditional generation of prompts, effectively replacing costly discrete optimization with a small number of parallelizable samples. We provide a probabilistic analysis demonstrating that under mild fidelity assumptions, only a few conditional samples are required to recover high-reward (harmful) prompts. Empirically, our method substantially outperforms prior attacks in both success rate and compute cost, producing low-perplexity, diverse jailbreaks that transfer to a wide range of black-box target models, including robustly trained and proprietary LLMs. Beyond adversarial prompting, our framework opens new directions for red teaming, automated prompt optimization, and leveraging emerging Flow- and Diffusion-based LLMs.
David Lüdke, Tom Wollschläger, Paul Ungermann +2
Department of Computer Science & Munich Data Science Institute Technical University of Munich, Germany