cs.CLNov 15, 2026

Risk-certified prompt optimization for LLM-as-a-judge guardrails

Authors: Hussein Jawad, Alexandre Landeau, Guy Fleury Iradukunda, Paul Beaumont, Théodore Rezel, Sofiane Schaack, Nicolas J-B. Brunel

Organizations: Universit´e d’´Evry Val d’Essonne · Capgemini · CentraleSup´elec · ´Ecole Nationale Sup´erieure d’Informatique pour l’Industrie et l’Entreprise (ENSIIE)

Abstract

LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level α with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on heldout tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

    Mar 26, 2026Hieu Xuan Le, Benjamin Goh, Quy Anh TangPrompt Injection DetectionLLM Guardrails

  2. System Prompts Affect Response Bias More Than Discriminability in LLM Safety

    Nov 8, 2026Mohan W. Gupta, Sandy TanwisuthLLM Safety EvaluationRefusal Behavior in Language Models

  3. kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail

    Jul 2, 2026Mahmoud Abdelfattah, Hamid Nasiri, Peter GarraghanLLM GuardrailsSafety Filtering