C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails

Mintong Kang; Zhaorun Chen; Bo Li

cs.AIOct 26, 2026

C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails

Authors: Mintong Kang, Zhaorun Chen, Bo Li

Organizations: UIUC · UChicago · UIUC & UChicago

Abstract

Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they can still be effectively jailbroken. Off-the-shelf guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, existing efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject to safety constraints. Furthermore, we introduce a generation risk certification framework and derive a high-probability upper bound on the safety risk of the proposed CSD algorithm. We prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data.

Figures & tables

Figure 1: C-SafeGen comprises (I) a claim-based stream decoding algorithm (CSD) and (II) a safety risk certification framework. The CSD algorithm utilizes an off-the-shelf guardrail model $G$ to monitor sequentially generated claims, flagging unsafe claims when they exceed a predefined threshold $\alpha_{b}$ . When such an event occurs, the unsafe claim is backtracked and resampled. Additionally, CSD employs auxiliary safety decoding strategies to enable safe resampling if claim generation stagnates beyond a tolerance threshold $\alpha_{g}$ . The safety risk certification framework provides two types of guarantees: (1) Given a CSD configuration $\alpha$ , the generated output is guaranteed to achieve a conformal safety risk $r_{\alpha}$ , as established in Theorem 1 ; (2) If the goal is to attain an expected safety risk $r$ , any CSD configuration within the set $\bm{\alpha}_{r}$ , as specified in Theorem 2 , can ensure this objective.

Panel of Figure 2 — Figure 2: Evaluation of conformal safety risk (upper bound from Theorem 1 ) and empirical safety risk (mean risk on sampled test sets) across two benchmarks—AdvBench and JailbreakBench—using Llama-3.1 models (8B and 13B) with ShieldGemma-9B as the guardrail. Results show that (1) conformal risk bounds are valid and tight; and (2) our CSD method consistently achieves the lowest safety risk. Note: Grey bars result from overlapping dots.

Panel of Figure 3 — Figure 3: Evaluation of SafeGen configurations with nominal safety risk $0.10$ ( Theorem 2 ). Valid configurations remain below the nominal risk, and larger backtrack thresholds $\alpha_{b}$ yield lower and more stable risks. Note: The grey bar results from overlapping grey dots.

Figure 4: Evaluation of decoding runtime and mean safety risk for the strongest baseline Best-of-N, our proposed SafeGen protocol CSD, and CSD with KV cache. CSD achieves a better efficiency-safety tradeoff than Best-of-N. KV cache preserves safety while significantly reducing decoding time.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 442 papers

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
Aug 11, 2026Xinzhe Huang, Biwu Yao, Kedong Xiu +4LLM Guardrails Language Model Calibration
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China +3
BarrierSteer: LLM Safety via Learning Barrier Steering
Feb 23, 2026Thanh Q. Tran, Arun Verma, Kiwan Wong +3LLM Safety Adversarial Robustness of Language Models
Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre, Republic of Singapore · CSAIL, Massachusetts Institute of Technology, USA +1
DNAlign: Dynamic Null-Space Safe Alignment for LLMs
Oct 2, 2026Jisheng Dang, Yushuo Zhao, Dewei Liu +6LLM Alignment LLM Safety Alignment
School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China

Aug 11, 2026J/K move · Enter open · S save

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

Xinzhe Huang, Biwu Yao, Kedong Xiu, Mengnan Zhao+3

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China+3

LLM Guardrails Language Model Calibration Safety Filtering LLM Safety Evaluation LLM Uncertainty Estimation LLM Safety Monitoring

Open paper AskPDF arXiv

Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail ProbGuard to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, ProbGuard achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6% and 71.9%, respectively, over the best baseline. ProbGuard further limits the attack success rate to at most 1% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.

Abstract

LLM Safety

Explore similar work

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

BarrierSteer: LLM Safety via Learning Barrier Steering

DNAlign: Dynamic Null-Space Safe Alignment for LLMs