Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades

Edoardo Pona; Milad Kazemi; Mehran Hosseini; Yali Du; David Watson; Osvaldo Simeone; Nicola Paoletti

cs.LGApr 15, 2026

Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades

Authors: Edoardo Pona, Milad Kazemi, Mehran Hosseini, Yali Du, David Watson, Osvaldo Simeone, Nicola Paoletti

Organizations: King’s College London · University of Manchester · Northeastern University London

Abstract

Monitoring LLM safety at scale requires balancing cost and accuracy: a cheap latent-space probe can screen every input, but hard cases should be escalated to a more expensive expert. Existing cascades delegate based on probe uncertainty, but uncertainty is a poor proxy for the utility of an expert call, as it ignores whether the expert would actually improve the prediction. To address this problem, we introduce Calibrate-Then-Delegate (CTD), a model-cascade approach that provides probabilistic guarantees on either computation cost or safety performance while enabling instance-level (streaming) decisions. CTD builds on a novel delegation value (DV) probe, a lightweight model operating on the same internal representations as the safety probe that predicts the benefit of replacing the probe score with the expert score after an expert call. CTD calibrates a threshold on the DV signal using held-out data and multiple hypothesis testing, yielding finite-sample guarantees while optimising the complementary objective. Evaluated on four safety datasets using learned-attention probes, CTD improves over uncertainty-based delegation over much of the budget range, avoids unnecessary expert calls, and extracts positive-value subsets even from groups where the expert score is worse than the probe score on average. Ablations across four probe architectures show that these findings are not specific to any particular architecture.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Calibrate-Then-Delegate (CTD) for safety monitoring. The safety probe scores every input. At inference time (top), the expert $\epsilon$ is invoked if the DV probe exceeds a threshold $\lambda$ ; routing is separate from score merging. The calibration diagram (bottom) shows the budget-constrained mode, which guarantees $R_{\text{B}}$ while optimising $R_{\text{P}}$ . The performance-constrained mode reverses these roles.

	Score	Outcome
Safety probe $\rho(x)$	$\vphantom{10^{-4}}0.051$	low-stakes ${\color[rgb]{0,0.6,0}\checkmark}$
Expert $\epsilon(x)$	${\approx}1.00$	high-stakes ${\color[rgb]{0.6,0,0}\times}$
Unc. top- $k$	rank $24/128$	delegates ${\color[rgb]{0.6,0,0}\times}$
DV probe $d(x)$	$-0.66$	retains ${\color[rgb]{0,0.6,0}\checkmark}$
Label: low-stakes, $v(x,y)=-0.95$

Figure 2: Two failures of uncertainty-based routing with a strong expert (

\lambda=0.39

\alpha_{\mathrm{B}}=0.20

B=128

). Left : an unnecessary expert call on MTSamples. Right : a missed beneficial call on Anthropic HH.

Figure 3: Cascade performance vs. target delegation budget with learned-attention safety and DV probes and a strong expert (Gemma-3-27B-IT,

B\!=\!128

); curves show the mean over three seeds. Left : AUROC, with Pareto filtering optimising AUROC. Right : accuracy, with Pareto filtering optimising accuracy. CTD, DV top-

k

, and Oracle top-

k

use expert overwrite; the uncertainty methods average probe and expert scores, following McKenzie et al. (2025) . Calibrated thresholds are solid and top-

k

strategies dashed; dotted and dashed grey lines show probe-only and expert-only performance.

Figure 4: Same as Figure 3 but with a weak expert (Llama-3.2-1B-Instruct). The left panel optimises AUROC and the right panel optimises accuracy. Expert-only performance (dashed grey) is below probe-only performance (dotted grey). Averaging hedges the uncertainty baseline against harmful expert overwrite, but still spends its fixed quota. CTD instead caps delegation and plateaus at substantially higher performance.

Figure 5: Weak-expert comparison under a common overwrite rule with learned-attention probes. At

\alpha_{\mathrm{B}}=1

, each fixed-quota top-

k

method delegates every input and converges to expert-only performance, illustrating harmful over-delegation. For Unc. top-

k

, overwrite replaces the averaging rule of McKenzie et al. (2025) .

Figure 6: Realised delegation rate versus target budget for CTD with attention probes. Curves and bands show the mean and one standard deviation over three seeds. Policies are selected separately using the AUROC and accuracy objectives. Fixed-quota top-

k

spends the complete target budget (dotted diagonal); CTD can leave budget unused when more aggressive policies are empirically dominated.

	Strong expert (Gemma-3-27B-IT)					Weak expert (Llama-3.2-1B-Instruct)
	Mean $v(x,y)$			Rate		Mean $v(x,y)$			Rate
Group	All	CTD	Unc	CTD	Unc	All	CTD	Unc	CTD	Unc
Anthropic HH	$+0.238$	$+0.470$	$+0.485$	$31.8\%$	$11.4\%$	$-0.147$	$+0.233$	$+0.140$	$25.5\%$	$11.4\%$
MTSamples	$+0.102$	$+0.381$	$+0.122$	$8.0\%$	$14.5\%$	$-0.182$	$+0.057$	$-0.230$	$9.9\%$	$14.5\%$
MTS-Dialog	$+0.140$	$+0.253$	$+0.334$	$31.5\%$	$11.3\%$	$-0.290$	$+0.112$	$-0.048$	$11.8\%$	$11.3\%$
ToolACE	$+0.093$	$+0.302$	$+0.163$	$7.5\%$	$35.3\%$	$-0.098$	$+0.231$	$+0.072$	$18.9\%$	$35.3\%$

Table 1: Group-conditional results at

\alpha_{\mathrm{B}}=0.20

with learned-attention probes, averaged over three seeds. Columns report mean

v(x,y)

for all examples and the CTD- and uncertainty-selected subsets, together with their delegation rates. Green and red shading denote positive and negative mean DV.

Figure 7: Empirical validation of a performance-risk guarantee with learned-attention probes and the strong expert over 500 random splits (

\alpha_{\mathrm{P}}=0.20

, i.e. accuracy at least 0.80;

\delta_{\mathrm{P}}=0.10

). CTD uses LTT hypothesis testing; the empirical selector uses identical data and candidates without testing. The histogram conditions on returning a policy. Weak-expert results are in the supplementary material.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Safety probe		DV Spearman correlation
Configuration	AUROC	Accuracy	Strong expert	Weak expert
Mean+Ridge	$.810\pm.009$	$.702\pm.014$	$.272\pm.004$	$.503\pm.012$
Attention+Attention	$\mathbf{.848\pm.020}$	$\mathbf{.728\pm.025}$	$.329\pm.011$	$.571\pm.028$
Softmax+Softmax	$.810\pm.013$	$.585\pm.010$	$\mathbf{.497\pm.017}$	$.585\pm.009$
MLP+MLP	$.811\pm.015$	$.699\pm.007$	$.303\pm.011$	$\mathbf{.597\pm.009}$

Appendix

Table 2: Safety- and DV-probe performance across architectures (mean

\pm

standard deviation over three seeds). DV quality is measured by rank correlation with the continuous delegation value. The best result in each column is bold.

Figure 8: Cross-architecture comparison of CTD over the complete target-budget sweep at

B\!=\!128

. Rows show the strong and weak experts. Left-column policies optimise AUROC; right-column policies optimise accuracy. Curves and bands are the mean and one standard deviation over three training seeds. Except for Mean, which denotes Mean+Ridge, each label means that the same architecture is used for both probes. The expert-only baseline is shown for reference.

Figure 9: Fixed score-merging controls with learned-attention probes and the strong expert. Top : every method overwrites the probe score on delegated inputs. Bottom : every method averages the probe and expert scores. The left panel in each row optimises AUROC for calibrated policies; the right panel optimises accuracy. Curves show means over three seeds.

Figure 10: Fixed score-merging controls with learned-attention probes and the weak expert. Under overwrite, poorly routed expert calls can directly harm predictions; averaging hedges this effect but attenuates the benefit of selective expert corrections.

Figure 11: Mean

v(x,y)

of the top-

k

examples ranked by each signal for learned-attention probes, as a function of selection fraction

k/N

. (Left) : strong expert. (Right) : weak expert. Lines and bands show the mean and one standard deviation over three seeds. The ground-truth ranking provides an upper bound.

	Strong expert (Gemma-27B)				Weak expert (Llama-1B)
	Binary		Continuous		Binary		Continuous
$\alpha_{\mathrm{B}}$	AUC	Acc	AUC	Acc	AUC	Acc	AUC	Acc
10%	.849	.757	.856	.751	.834	.747	.830	.737
20%	.870	.788	.874	.783	.834	.757	.835	.759
30%	.880	.810	.883	.808	.834	.763	.849	.780
50%	.885	.830	.897	.830	.825	.739	.851	.777

Appendix

Table 3: DV top-

k

cascade performance at

B=128

for binary vs. continuous delegation value targets. Continuous DV consistently matches or improves AUC; the gain is largest at high budgets and with the weak expert.

Figure 12: Difference in DV top-

k

performance (continuous

-

binary) vs. delegation budget for three batch sizes. Top : strong expert (Gemma-27B). Bottom : weak expert (Llama-1B). Positive values indicate that the continuous formulation outperforms the binary one. The advantage grows with the budget, consistent with the finer-grained ranking provided by the continuous target.

Figure 13: Mean+Ridge configuration with the strong expert (Gemma-3-27B-IT): cascade performance at

B\!=\!32

(top) and

B\!=\!64

(bottom).

Figure 14: Mean+Ridge configuration with the weak expert (Llama-3.2-1B-Instruct): cascade performance at

B\!=\!32

(top) and

B\!=\!64

(bottom).

Figure 15: Distribution of realised delegation rates across 500 random calibration splits (

\alpha_{\mathrm{B}}=0.3

\delta_{\mathrm{B}}=0.1

). Left : strong expert (Gemma-3-27B-IT). Budget-only LTT violates at 8.6%, close to

\delta_{\mathrm{B}}

; Pareto LTT is more conservative at 3.0%. Right : weak expert (Llama-3.2-1B-Instruct). Both methods stay below

\delta_{\mathrm{B}}

(6.6% and 4.0%), with the Pareto filter adapting its conservatism to the limited effective delegation capacity.

Figure 16: Performance-risk guarantee validation across probe architectures over 500 random splits (

\delta_{\mathrm{P}}=0.10

). Rows show probe architectures; columns show strong and weak experts. Dashed red lines mark accuracy

1-\alpha_{\mathrm{P}}

. Histograms condition on selection, while titles report unconditional violation and selection rates.

Figure 17: Realised delegation rate versus guaranteed minimum accuracy for the four probe architectures. Left : strong expert. Right : weak expert. Points and error bars show the mean and one standard deviation over three training seeds among certified policies.

Figure 18: Mean+Ridge cascade performance with AUROC error as the Pareto-testing objective in both panels (strong expert,

B\!=\!128

Figure 19: Mean+Ridge cascade performance with AUROC error as the Pareto-testing objective in both panels (weak expert,

B\!=\!128

LLM Safety Evaluation

Large language model safety evaluation measures harmful compliance, robustness, and safe utility across prompts, tools, and real-world tasks.

See all 258 papers

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
Aug 11, 2026Xinzhe Huang, Biwu Yao, Kedong Xiu +4LLM Guardrails Language Model Calibration
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China +3
Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs
Jun 9, 2026Huizhen Shu, Xuying Li, Piao XueLLM Guardrails Safety Filtering
ModelOneAI · yunshanai
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiSafety Filtering LLM Safety
Wrynx Inc. · Independent Researcher

Aug 11, 2026J/K move · Enter open · S save

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

Xinzhe Huang, Biwu Yao, Kedong Xiu, Mengnan Zhao+3

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China+3

LLM Guardrails Language Model Calibration Safety Filtering LLM Safety Evaluation LLM Uncertainty Estimation LLM Safety Monitoring

Open paper AskPDF arXiv

Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail ProbGuard to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, ProbGuard achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6% and 71.9%, respectively, over the best baseline. ProbGuard further limits the attack success rate to at most 1% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.

Abstract

LLM Safety Evaluation

Explore similar work

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models