We prove that no continuous, utility-preserving wrapper defense-a function D:X→X that preprocesses inputs before the model sees them-can make all outputs strictly safe for a language model with connected prompt space, and we characterize exactly where every such defense must fail. We establish three results under successively stronger hypotheses: boundary fixation-the defense must leave some threshold-level inputs unchanged; an ε-robust constraint-under Lipschitz regularity, a positive-measure band around fixed boundary points remains near-threshold; and a persistent unsafe region under a transversality condition, a positive-measure subset of inputs remains strictly unsafe. These constitute a defense trilemma: continuity, utility preservation, and completeness cannot coexist. We prove parallel discrete results requiring no topology, and extend to multi-turn interactions, stochastic defenses, and capacity-parity settings. The results do not preclude training-time alignment, architectural changes, or defenses that sacrifice utility. The full theory is mechanically verified in Lean 4 and validated empirically on three LLMs.
Figures & tables
Result
Target
Method
Impossibility
Verified
Tsipras et al. [ 21 ]
Classifier
PAC / Gaussian
Robustness + accuracy
No
Fawzi et al. [ 11 ]
Classifier
Lipschitz geom.
Adv. invulnerability
No
Wolpert & Macready [ 22 ]
Optimizer
Counting
Universal dominance
No
Cohen et al. [ 5 ]
Smoothed clf.
Rand. smoothing
(Certifies bound)
No
This work
Def. wrapper
Topology
Cont. + util. + compl.
Lean 4
Empirical findings consistent with our derived predictions
Table 1: Positioning among impossibility results (top) and empirical findings independently consistent with our predictions (bottom).
Figure 1: Schematic of the prompt space. The defense D must leave all safe inputs unchanged (utility preservation) and tries to remap unsafe inputs into the safe region. Boundary fixation ( Theorem 4.1 ) proves the defense must also leave some boundary points unchanged, the black dot z passes through with no remediation.
Figure 2: The defense trilemma. Any continuous wrapper defense on a connected space can satisfy at most two of the three properties. The bottom edge is our main result; the other two edges correspond to counterexamples in Appendix C .
Figure 3: The three impossibility results on a 1D cross-section. (a) The defense must fix boundary point z (Thm. 4.1 ). (b) Near z , Lipschitz regularity constrains the defense to a shallow ε -band (yellow; Thm. 5.1 ). (c) Where f rises faster than the defense budget ℓ(K+1)δ (dashed red), the region above τ persists (red shading; Thm. 6.3 ).
Theorem
Predicts
Confirmed by
Basin Structure ( A.1 )
Basins are open with positive measure
Heatmaps show extended regions
Fragmentation ( A.2 )
Smooth → large basins; rough → mosaic
Llama: mesa; GPT-OSS: mosaic
Convergence ( D.2 )
Attacks exhibit monotone convergence
Convergence curves plateau
Transferability ( D.3 )
Similar surfaces → shared basins
Llama .93→ GPT-OSS .73→ Mini .47
Authority ( D.4 )
Horizontal banding
Bands at a2≈.25 – .35 , .65 – .85
Persistent ( 6.3 )
Steep boundaries → unsafe volume persists
Llama’s .93 plateau persists under defense
Table 2: Falsifiable predictions confirmed by empirical data.
Target
τ
Filled
L
G
∣Spred∣
∣Sact∣
TP
FP int
FN
gpt-5-mini-2025-08-07
0.30
14
6.13
6.13
7
7
7
0
0
gpt-3.5-turbo-0125 (smoke)
0.50
9
11.44
11.44
7
7
7
0
0
gpt-3.5-turbo-0125 (saturated)
0.50
82
23.63
23.63
66
66
66
0
0
Table 3: Live empirical validation of Theorem 6.3 against the canonical continuous defense ( D=id , K=1 , ℓ=0 ). ∣Spred∣ is the predicted steep-set size {x:f(x)>τ+ℓ(K+1)dist(x,z∗)} ; ∣Sact∣ is the actual post-defense persistent set {x:f(D(x))>τ} ; TP/FP int /FN are the per-cell confusion-matrix counts. Across all three live runs, the two sets match cell-for-cell, with zero interior false positives.
Figure 4: Non-vacuous validation of Theorem 6.3 on the saturated gpt-3.5-turbo-0125 grid ( 82 filled cells, τ=0.5 ) under the GP-smooth oblique defense ( θ=89.5∘ , ℓ=0.86 , K=1.05 ). Left: the steep set Ssteep={x:f(x)>τ+ℓ(K+1)d(x,z∗)} ; ∣Ssteep∣=3 . Right: cells that actually remain unsafe after defense, Sact={x:f(D(x))>τ} ; ∣Sact∣=68 . Transversality holds non-trivially ( G=23.6≫ℓ(K+1)=1.77 ) and the defense is genuinely non-identity ( 82 cells displaced). All 3 predicted-persistent cells are indeed actually persistent (TP =3 , FP int=0 ). The 65 false negatives (red on the right, green on the left) are not theorem violations: the theorem predicts Ssteep⊆Sact , not equality. The 8 boundary cells (black outlines) are identified on the GP-smoothed surface.
Defense
K
ℓ
∣Spred∣
∣Sact∣
TP
FP int
FN
identity
1.00
0.00
66
66
66
0
0
bounded_step(1)
3.61
23.63
1
55
0
0
55
bounded_step(2)
3.61
23.63
1
47
0
0
47
bounded_step(3)
4.00
23.63
1
29
0
0
29
bounded_step(5)
8.60
23.63
1
3
0
0
3
bounded_step(8)
8.60
23.63
1
0
0
0
0
Table 4: Defense sweep on the saturated gpt-3.5-turbo-0125 grid ( 82 filled cells, τ=0.5 ). Empirical defense Lipschitz constant K grows with reach; the steep-set size shrinks ( bounded_step is non-continuous, so its steep set collapses to the anchor under the discrete check) but the actual persistent set falls cleanly along the K-tradeoff curve from Theorem 7.3 . Across all 7 defenses, FP int=0 .
Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cost. Rather than treating defenses as a single class, we organize them by operational strategy and examine how different strategies correlate with different side-effect profiles. Across state-of-the-art defense methods, widely used benchmark datasets, and representative open-source LLMs, we find that defenses rarely improve downstream capability, but instead vary in how they trade safety gains against usability and efficiency. In particular, rule-based defenses best preserve task performance, highly conservative self-reflective defenses often increase over-refusal, and multi-round defenses incur the largest runtime overhead. These results provide both a benchmark for evaluating defense side effects and practical guidance for selecting defenses under deployment constraints.
Tong Zhang, Zexin Li, Simin Chen +1
Fudan University · University of California, Riverside · Columbia University
Large language models (LLMs) perform remarkably well on complex tasks, yet remain highly vulnerable to prompt injection attacks, where malicious instructions embedded in external data can override user intent. Existing defenses remain limited by model fine-tuning requirements, vulnerability to adaptive attacks, or reliance on brittle handcrafted prompts. We argue that a fundamental source of this vulnerability is the lack of an explicit representation of trust provenance. To address this, we introduce Learnable Trust-Boundary Delimiters (LTBD), a lightweight defense that explicitly encodes trust boundaries in the input while keeping the LLM parameters unchanged. LTBD uses a small number of learnable delimiters to distinguish trusted user instructions from untrusted external data, enabling the model to better respect the intended trust hierarchy. Experimental results show that LTBD substantially outperforms inference-time defenses and performs competitively with training-based approaches, while preserving benign-task utility and introducing negligible inference overhead. In particular, LTBD achieves 0.00% ASR on AlpacaFarm and only 0.11-0.19% ASR on TaskTracker. LTBD also remains effective under adaptive attacks, where adversaries have full knowledge of the defense and explicitly attempt to bypass it.
Luman Zhao, Minghui Xu, Yue Zhang +1
Shandong University, Shandong Academy of Artificial Intelligence
Neuron- and path-level interventions offer the finest-grained route to defending large language models (LLMs) against jailbreak attacks, yet existing methods fall short of this promise, i.e., they often compromise model utility significantly. Specifically, one line of work suppresses toxic neurons to erase harmful semantics, but since such semantics are distributed across the network, blocking every pathway forces a large intervention footprint. An alternative line of research focus on identify safety neurons using external classifiers. While promising, the existing approaches suffer from compromising neurons that are important for the model utility as well. Moreover, both approaches remain always on and thus perturb every benign request even when no attack is present. To address these limitations, we present \ours{}, a training-free defense that first identifies safety-specific neurons through per-neuron hypothesis tests under false-discovery-rate control together with a utility-specificity filter. Based on this identification, a trigger-style clamp holds the selected neurons at their harmful-conditional mean activations, injecting an internal harmful-input signal that triggers the refusal behavior learned during alignment. The clamp is then realized by two provably equivalent deployment modes, namely a detector-gated inference-time intervention and an offline bias-patch weight edit. Extensive experiments across four safety-aligned LLMs and four representative attacks demonstrate that \ours{} reduces the average attack success rate to at most 2.0% while incurring a utility drop of only 0.5% to 5.3% on MT-Bench, the smallest among all defenses. Code is available at https://anonymous.4open.science/r/Tripwire-65C4.