cs.AIOct 26, 2026

C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails

Authors: Mintong Kang, Zhaorun Chen, Bo Li

Organizations: UIUC · UChicago · UIUC & UChicago

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they can still be effectively jailbroken. Off-the-shelf guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, existing efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject to safety constraints. Furthermore, we introduce a generation risk certification framework and derive a high-probability upper bound on the safety risk of the proposed CSD algorithm. We prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data.

Figures & tables

Figure 1: C-SafeGen comprises (I) a claim-based stream decoding algorithm (CSD) and (II) a safety risk certification framework. The CSD algorithm utilizes an off-the-shelf guardrail model GG to monitor sequentially generated claims, flagging unsafe claims when they exceed a predefined threshold αb\alpha_{b} . When such an event occurs, the unsafe claim is backtracked and resampled. Additionally, CSD employs auxiliary safety decoding strategies to enable safe resampling if claim generation stagnates beyond a tolerance threshold αg\alpha_{g} . The safety risk certification framework provides two types of guarantees: (1) Given a CSD configuration α\alpha , the generated output is guaranteed to achieve a conformal safety risk rαr_{\alpha} , as established in Theorem 1 ; (2) If the goal is to attain an expected safety risk rr , any CSD configuration within the set αr\bm{\alpha}_{r} , as specified in Theorem 2 , can ensure this objective.
Figure 2: Evaluation of conformal safety risk (upper bound from Theorem 1 ) and empirical safety risk (mean risk on sampled test sets) across two benchmarks—AdvBench and JailbreakBench—using Llama-3.1 models (8B and 13B) with ShieldGemma-9B as the guardrail. Results show that (1) conformal risk bounds are valid and tight; and (2) our CSD method consistently achieves the lowest safety risk. Note: Grey bars result from overlapping dots.
Figure 3: Evaluation of SafeGen configurations with nominal safety risk 0.100.10 ( Theorem 2 ). Valid configurations remain below the nominal risk, and larger backtrack thresholds αb\alpha_{b} yield lower and more stable risks. Note: The grey bar results from overlapping grey dots.
Figure 4: Evaluation of decoding runtime and mean safety risk for the strongest baseline Best-of-N, our proposed SafeGen protocol CSD, and CSD with KV cache. CSD achieves a better efficiency-safety tradeoff than Best-of-N. KV cache preserves safety while significantly reducing decoding time.
More in this topic

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 442 papers

Explore similar work

CardsList
Aug 11, 2026

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail ProbGuard to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, ProbGuard achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6% and 71.9%, respectively, over the best baseline. ProbGuard further limits the attack success rate to at most 1% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.

Xinzhe Huang, Biwu Yao, Kedong Xiu +4

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China +3

LLM GuardrailsLanguage Model CalibrationSafety FilteringLLM Safety EvaluationLLM Uncertainty EstimationLLM Safety Monitoring
AskPDF
Feb 23, 2026

BarrierSteer: LLM Safety via Learning Barrier Steering

Despite the strong performance of large language models (LLMs) across diverse tasks, their susceptibility to adversarial attacks and unsafe content generation remains a significant obstacle to deployment, particularly in high-stakes settings. Addressing this challenge requires safety mechanisms that are both practically effective and theoretically grounded. In this paper, we introduce BarrierSteer, a novel inference-time framework that improves response safety by embedding learned nonlinear safety constraints directly into the model's latent representation space. BarrierSteer treats hidden-state safety classifiers as Control Barrier Functions (CBFs), enabling constraint-guided steering of unsafe latent trajectories during generation. By composing multiple safety constraints through efficient constraint merging without modifying the underlying LLM parameters, BarrierSteer preserves model utility. We provide theoretical results showing that applying CBFs in the latent space yields a principled, modular, and computationally efficient approach for steering with respect to learned safety constraints, with guarantees conditional on the learned barriers capturing the intended safety property. Our extensive experimental results across multiple model families and datasets demonstrate that BarrierSteer substantially reduces adversarial attack success rates and unsafe generations, outperforming the existing method. The code is available in our GitHub repository.

Thanh Q. Tran, Arun Verma, Kiwan Wong +3

Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre, Republic of Singapore · CSAIL, Massachusetts Institute of Technology, USA +1

LLM SafetyAdversarial Robustness of Language ModelsControl Barrier FunctionsLatent Space Optimization
AskPDF
Oct 2, 2026

DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.

Jisheng Dang, Yushuo Zhao, Dewei Liu +6

School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China

LLM AlignmentLLM Safety AlignmentInference-Time Language Model Alignment
AskPDF