cs.LGNov 16, 2026

Tight PAC-Bayes Generalisation Guarantees for Large Language Model Judges

Authors: Tom A. Lamb, Philip H. S. Torr, Tim G. J. Rudner

Organizations: University of Oxford · University of Toronto, Vector Institute, Vijil

Abstract

Large language model (LLM) judges are increasingly used to monitor the outputs of LLM-based AI agents, yet formal guarantees on the reliability of these judges remain largely absent, limiting confidence in their use for safety-critical oversight. We consider certifying such a judge and define its risk over safety classifications of prompt--response pairs drawn from a deployed generator. We apply a PAC-Bayes framework to this setting, refining existing compression-based methodology with the use of tighter bounds and more efficient coding. We show that standard parameter-efficient fine-tuning can yield non-vacuous and even tight certificates for safety judges, for both classification and Brier risk, without holding out data for validation. We introduce LoRA-GT, which shares a single quantisation scale across layers and achieves the tightest certificates with minimal degradation in predictive performance. Across five guardrail models and four benchmarks, we find that certificate tightness is governed more by adaptation complexity than by base model scale. Finally, we show how far quantisation can be pushed before the resulting increase in risk outweighs the bound tightening gained from shorter descriptions. Together, these results provide a practical route toward certifiable scalable oversight and offer insight into how the complexity of safety adaptations shapes generalisation.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision

    Jul 9, 2026Berkay AnahtarciLLM EvaluationPartial Identification

  2. Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

    Aug 18, 2026Sher Badshah, Ali Emami, Hassan SajjadLLM-as-a-JudgeSelective Prediction

  3. Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers

    May 11, 2026Nikita Kezins, Urbas Ekka, Pascal Berrang +1Neural Network VerificationLanguage Model Safety Evaluation