Risk-certified prompt optimization for LLM-as-a-judge guardrails
Authors: Hussein Jawad, Alexandre Landeau, Guy Fleury Iradukunda, Paul Beaumont, Théodore Rezel, Sofiane Schaack, Nicolas J-B. Brunel
Organizations: Universit´e d’´Evry Val d’Essonne · Capgemini · CentraleSup´elec · ´Ecole Nationale Sup´erieure d’Informatique pour l’Industrie et l’Entreprise (ENSIIE)
LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level α with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on heldout tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates.
Figure 2: CRISP: adaptive prompt search with reusable-holdout risk certification.
LLM-as-a-judge guardrail
Method
BIPIA
Deepset
Risk ↓
Utility ↑
Risk ↓
Utility ↑
–
Baseline classifier guardrails
ProtectAI
0.88
0.88
0.55
0.99
PIGuard
0.15
0.86
0.08
1.00
Prompt Guard
0.13
0.78
0.07
0.14
Model Armor
0.94
0.98
0.72
0.98
gpt-4o-mini
Baseline prompt optimization
Naive
0.60
0.90
0.29
0.92
Table 1: Prompt-injection test results. Target risk levels: α∈{0.10,0.20,0.30} .
Method
SPL (leakage)
SID (BeaverTails)
Risk ↓
Utility ↑
Risk ↓
Utility ↑
Baseline prompt optimization
Naive
0.17
0.38
0.52
1.00
ProTeGi
0.15
0.78
0.13
0.93
GEPA
0.17
0.58
0.12
0.95
MIPROv2
0.06
0.65
0.13
0.97
Ours ( α=0.3 )
CPS
0.21
0.78
0.15
0.97
Table 2: Output-guardrail test results
Figure 3: Best-of- N jailbreaking on BIPIA. Missed-attack risk as a function of the attacker query budget N , using gpt-4o-mini as the judge.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Access
Sources
Judge inputs
BIPIA
public
BIPIA ( Yi et al., 2025 )
context
Deepset
public
deepset/prompt-injections
user prompt
SPL
mixed
gabrielchua/system-prompt-leakage + Promptfoo
system prompt, response
SID
public
BeaverTails ( Ji et al., 2023 )
protected items, request, response
Appendix
Table 3: Guardrail benchmarks used in Section 5 . SPL is a constructed mix, not a fully public release.
Dataset
Train n (mal/ben)
Cert. n (mal/ben)
Test n (mal/ben)
BIPIA
600 (301/299)
1800 (903/897)
597 (294/303)
Deepset
132 (58/74)
397 (149/248)
132 (55/77)
SPL
93 (53/40)
278 (140/138)
92 (52/40)
SID
599 (299/300)
1798 (889/909)
597 (306/291)
Appendix
Table 4: Split sizes used in the experiments. Test counts are after excluding API content-filter rejections (3 examples on BIPIA, 3 on SID, none on Deepset or SPL).
T
Certified
Calib Risk
Calib Utility
Test Risk
Test Utility
Candidates
Rewrites
0
No (0/1)
–
–
–
–
1
0
1
Yes (1/2)
0.01
0.48
0.01
0.50
2
1
2
Yes (3/4)
0.06
0.68
0.06
0.66
4
3
5
Yes (7/10)
0.13
0.79
0.15
0.82
10
9
10
Yes (14/20)
0.12
0.81
0.11
0.83
20
19
20
Yes (21/40)
0.12
0.81
0.11
0.83
40
39
Appendix
Table 5: CRISP search horizon on BIPIA at α=0.20 . Certified indicates whether the certified set is nonempty, with the number of certified candidates relative to the total pool size shown in parentheses. Calib denotes Dcert , while Test denotes the filtered test set. “–” indicates that no candidate is certified. The same prompt is selected at T=10 , T=20 , and T=40 .
T
Certified
Calib Risk
Calib Utility
Test Risk
Test Utility
Candidates
Rewrites
0
No (0/1)
–
–
–
–
1
0
1
Yes (1/2)
0.13
0.52
0.10
0.56
2
1
2
Yes (2/4)
0.13
0.52
0.10
0.56
4
3
5
Yes (5/10)
0.16
0.83
0.17
0.80
10
9
10
Yes (11/20)
0.20
0.85
0.24
0.84
20
19
20
Yes (22/40)
0.20
0.85
0.24
0.84
40
39
Appendix
Table 6: CRISP search horizon on BIPIA at α=0.30 . The same prompt is selected at T=1 and T=2 , while another prompt is selected at both T=10 and T=20 .
α=0.20
α=0.30
Rtest≤α
49/49
50/50
Abstention rate ( CT=∅ )
1/50
0/50
Mean Rtest
0.09
0.15
Appendix
Table 7: CRISP on 50 independent BIPIA re-splits at α=0.20 and α=0.30 . The fraction of certified splits with Rtest≤α is compared to 1−δ=0.90 .