Conformal Risk Control

Ask about this topic

Overview

Conformal risk control calibrates a model’s decisions on held-out examples so that future errors, losses, or safety violations stay within a chosen risk budget under assumptions such as exchangeability. Methods include threshold and prediction-set calibration, learn-then-test procedures, e-value tests, and extensions for non-monotone losses, groups, and multiple risk measures. Applications range from selective prediction and language-model abstention to medical triage, robotics, and control, where guarantees can make deployment trade-offs explicit. A central challenge is that marginal guarantees may not protect subgroups or survive distribution shift, while stronger guarantees and limited calibration data can reduce coverage or make systems overly cautious.

Latest papers

CardsList

64 papers6 papers in the last four weeks, up 20% on the four weeks before. 0.0% of all new papers.

Nov 6, 2026

Region-aware error triage for medical image segmentation with groupwise conformal risk control

Segmentation masks are increasingly reviewed rather than redrawn, which requires knowing where a mask is likely wrong. Learned error maps rank pixels by risk, but the number of pixels to warn is chosen ad hoc, and calibrating one warning threshold over all error pixels may leave errors far from the predicted contour under-flagged. We propose RACE-Triage, a post-hoc error triage framework that leaves the segmentation mask unchanged. A risk network is trained on out-of-fold predictions from backbone-independent inputs. The predicted contour partitions each image into a boundary band and a non-boundary region, and conformal risk control is applied separately in each region to bound the expected fraction of missed errors among cases with errors there. On ISIC 2018 dermoscopy and BUSI breast ultrasound (where patient identifiers are unavailable, so independence across images is assumed), pooled control met its overall target but missed on average 31.8% and 36.4% of non-boundary errors. Regional control reduced these fractions to 19.3% and 20.9% and raised the recall of size-filtered false-negative components from 0.722 to 0.818 and from 0.611 to 0.763. The mean warned area grew by 1.6 and 4.9 percentage points, and its 95th percentile reached 23.5% and 38.5%. The effect held across three seeds and a second backbone, and area-matched heuristics did not reproduce it. On the external PH2^2 set, frozen thresholds exceeded both targets, whereas recalibration with 100 local images brought the mean over repeated splits below them. Regional calibration improves non-boundary error coverage at a measured increase in warned area.

Nishi Kanta Paul, Md Shihabul Islam Shovo, Hoileong Lee

NIMISHES Lab, Dhaka, Bangladesh · Faculty of Electronic Engineering and Technology, Universiti Malaysia Perlis, 02600 Arau, Perlis, Malaysia

Medical Image SegmentationConformal Risk Control
AskPDF
Nov 4, 2026

When Does Risk-Controlled Throughput Forecasting Actually Control Risk? A Reproduction Study on LEO Satellite Link

Risk-budgeted throughput forecasting converts a point forecast of LEO satellite capacity into a conservative bound whose overestimation rate is held at a user-specified budget, so that admission control can act on it safely. We reproduce the state of the art, BG-CFQS, on the three Starlink traces its authors used and on an independent fourth dataset, and report four findings. First, its risk guarantee is reproducible but conditional on the calibration split: under a random split we recover every published quantity, including risk-pass on 3 of 3 datasets, but under a contiguous temporal split, which is what a deployed terminal faces, it holds on 1 of 3. Second, independent of any split, the method cannot serve a risk budget below 0.15, the low end of its candidate quantile set. On all four datasets the achieved rate is identical at budgets of 0.05, 0.10 and 0.15, overshooting by 1.4x to 3.7x at the tightest. Third, static conformal calibration cannot improve a ranking, only shift an operating point; we confirm this exactly on latency spike detection, where online calibration improves AUPRC by up to 51%. Fourth, group-conditional online recalibration splits into two mechanisms. The online part replicates: it tracks the budget within 1% from 0.05 to 0.35, where static calibration is 16 to 21% off. The group-conditional part does not: on three of four datasets, conditioning on any available covariate, including measured serving-satellite geometry, is worse than not conditioning, and our attempt to predict which datasets benefit failed. What is new here is empirical, not algorithmic: we claim the findings, not the methods.

Mayan Sharma, Kriti Saini, Devansh Behl +2

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, Tamil Nadu, India · Nelco Network Products Ltd., India

Conformal Risk ControlConformal PredictionGroup-Conditional Conformal PredictionLEO Satellite Networks
AskPDF
Nov 4, 2026

CARM: Conformal Adaptive Risk Masking for Reliable Scientific Super-Resolution

Deep learning-based super-resolution (SR) can reconstruct fine-scale Earth-observation fields from coarse observations, but reconstruction accuracy alone does not provide a calibrated rule for identifying reliable spatial regions. Conformal Mask Calibration (CMC), built on conformal risk control, converts a spatial uncertainty score into a trust mask and controls the expected worst-case local distortion under exchangeability. However, its fixed threshold can become miscalibrated in Earth-observation streams affected by seasonality and episodic events. We propose CARM\textit{CARM} (Conformal Adaptive Risk Masking), a statistical framework for adaptive spatial risk control under temporal distribution shift. CARM maintains experts with heterogeneous update rates and aggregates their threshold proposals online using an asymmetric surrogate loss, balancing rapid adaptation to distribution changes against stability during relatively stationary periods. All updates are causal and use only mature feedback available after a fixed delay, allowing CARM to operate with any SR model that provides a spatial uncertainty score. We prove that the time-averaged mask-conditioned risk converges to at most αα at an O(1/T)O(1/T) rate, without distributional assumptions and for any choice of aggregation weights across experts. On a daily South China Sea chlorophyll-a\textit{a} SR stream, CARM reduces the empirical violation rate from 43.49%43.49\% to 9.38%9.38\% and the moving-average exceedance frequency from 47.29%47.29\% to 1.22%1.22\% at α=0.10α=0.10, while retaining a non-trivial trusted region. Experiments on ERA5 wind and OSTIA sea-surface temperature further show consistent relative improvements across multiple SR backbones.

Guangtao Zhang, Haonan Wu, Junxuan Gao +5

SandGold AI Research, China · The Hong Kong University of Science and Technology (Guangzhou), China · Nanyang Technological University, Singapore +1

Adaptive Conformal InferenceConformal Risk ControlTemporal Distribution ShiftRemote Sensing Image Super-Resolution
AskPDF
Nov 4, 2026

Risk Control Certifies Rates, Not Magnitudes: An Audit of Selective Fetal Biometry

Prenatal ultrasound biometry must return a head circumference (HC) and refuse when the image cannot support one. Selective prediction trades coverage against error, but its finite-sample guarantees bound the rate of gross errors, not their magnitude. We audit a tool-grounded biometry pipeline on two separately pre-registered four-way splits of the 999 labelled HC18 images, retraining the segmenter three times on each. Calibrating the contour-completeness threshold by exact Clopper-Pearson risk control returns a certificate indistinguishable from a hand-set floor, yet the pipeline still emits a 271.3 mm error: a small, geometrically complete ring fitted to an image whose true HC is 324.6 mm. The accepted candidate scores 0.94 completeness, so the score cannot see the failure, and one outlier among ≈150\approx 150 answers leaves the certified risk bound (α=0.10α= 0.10) intact. A second, untouched split reproduces the mechanism on a different image (233.3 mm). Constraining the admissibility region---the anatomical range in which the measurand is defined---removes the family instead: raising the lower bound from 50 to 55 mm costs 0.4 coverage points and cuts the worst-case error on both splits to 11.4 and 11.1 mm, whereas suppressing it through the completeness threshold costs 19.0 and 12.1 points. At an anatomically defined 70 mm the frozen tests give 1.91 and 1.81 mm MAE.

Zhengshuai Yang, Linna Jiang, Changyun Li +1

College of Information and Intelligent Science and Technology, Hunan Agricultural University, Changsha 410128, China · National Clinical Research Center for Geriatric Disorders, Xiangya Hospital, Central South University, Changsha 410008, China

Selective PredictionUltrasound ImagingConformal Risk ControlUltrasound Image Segmentation
AskPDF
Nov 3, 2026

Weather Adaptive Autonomous Driving

Camera object detectors lose recall in rain, fog, snow and sand, and the confidence threshold at which a detector is deployed decides how many objects it misses and how many false alarms it raises. We study how a frozen detector should make this decision in adverse weather. We first reproduce SeqCRC, a conformal object detector whose first step calibrates the confidence threshold by conformal risk control (CRC), and use that step, calibrated separately in each weather condition, as our baseline. With YOLOv8x it keeps the held-out miss fraction within a 20% budget in every DAWN weather bin, where a single global threshold fails in fog, but the false alarms it needs are an output rather than a choice: from 1.6 to 117 per image across DAWN and BDD100K conditions. We then propose a calibrated trade-off dial. One setting, λλ, the number of false alarms accepted to avoid one missed object, selects the expected-cost-optimal decision rule: either one score threshold, or a per-detection rule that keeps a box when its calibrated probability of being a real object, estimated by gradient-boosted trees from the detector's own per-box outputs, is at least 1/(1+λ)1/(1+λ). F-score preferences map onto the same dial, and CRC applied to λλ itself is planned to restore a per-condition miss guarantee. The dial is fitted once on BDD100K train and graded only on BDD100K val and DAWN.

Nicholas Taylor, Alexander Picon, Sudip Dhakal

Dept. of Computing and Software Engineering, Florida Gulf Coast University, Fort Myers, FL, USA

Autonomous DrivingObject DetectionConformal Risk Control
AskPDF
Oct 31, 2026

Efficient Conformal Calibration for Paired Recommendation Decisions

Risk control can suppress useful changes to a frozen recommendation rule. We study how to recover valid activation from the same observed fractional losses. Two specified internal calibrators can lose efficiency by compressing losses into their sum or clipping auxiliary inverse counts before averaging. We propose Averaged Conformal Risk Calibration, or ACRC, which averages before clipping. A product identity proves expected-risk validity, while pointwise bounds characterize the additional activation. For fixed populations sampled without replacement, we connect activation to signed utility under heterogeneous users and switching constraints. Near the budgets, positive population mean gain and the stated nondegeneracy conditions give positive leading-order utility increments despite sampling dependence. Across six datasets, changing only calibration improves discounted net utility by 0.0102260.010226 over raw and 0.0083370.008337 over RB. A separately frozen Jester confirmation gives 0.011025±0.0010820.011025 \pm 0.001082 over raw with the rule and filter fixed, using a five-seed approximate 95 percent paired interval. Against information-matched SCoRE selecting risk or reward scores, the mean utility difference is +0.015934±0.001440+0.015934 \pm 0.001440. These gains use more mean realized risk within the declared budgets. Upstream failures delimit the benefit: efficient calibration releases useful modifications when the proposals themselves have value.

Xinyu Qiao, Yichen Lin, Zhewen Chen +2

Antai College of Economics and Management Shanghai Jiao Tong University, Shanghai, China · Meta

Conformal Risk ControlPost-Hoc CalibrationRecommender Systems
AskPDF
Oct 26, 2026

Macro Mondrian Conformal Risk Control

Conformal Risk Control (CRC) extends conformal prediction beyond the coverage setting to the control of general monotone loss functions, in a distribution-free and modelagnostic fashion. This makes it well-suited to problems with high-dimensional targets, such as the semantic segmentation task addressed in this work. However, CRC guarantees only marginal control, meaning that the risk is controlled on average over the data distribution. Under imbalanced distributions, the risk on inputs that contain rare objects may therefore remain uncontrolled, even when the marginal guarantee holds. We propose a new method that controls the risk on average over user-defined groups of the domain, drawing on Mondrian conformal prediction and macro-coverage algorithms. The method relies on a taxonomy, that is, a function that assigns a group to each data point, and reweights each point's risk by a coefficient that depends on its group probability, so that rare points contribute more than common ones. We illustrate the approach on two semantic segmentation datasets.

Julien Combes, Jean-Francois Coeurjolly, Alexandre Dervill{é}

SVH, Michelin · SVH · Michelin

Conformal Risk ControlConformal PredictionGroup-Conditional Conformal Prediction
AskPDF
Oct 14, 2026

Calibration and Risk-Budgeted Decisions with System One Models

LLM workflows and agents rely on a set of binary decisions like allow/block an input, or discard/retain a retrieved document. When human deferral is unavailable, uncertain cases must still receive an action. We study System One decision scores in this setting by defaulting to a conservative fallback when uncertain, and controlling the false omission risk within the subset of inputs assigned the utility-maximizing action. Across six datasets covering moderation, safety, and reranking, we compare general System One models with task baselines and a local open-weights fine-tuned System One model. We show that naive baselines for threshold selection exceed the target risk in roughly half of settings with defined test risk, compared with 7.8% for Learn Then Test (LTT), which leverages finite-sample statistical guarantees. Our results show why relying on a probability-like interface is insufficient for deployment decisions and how risk-budgeted selection trades useful action against observed error.

Soundouss Messoudi, Youness Mansar

Heudiasyc - UMR CNRS 7253 Université de Technologie de Compiègne · Independent Researcher France

LLM Decision-MakingSelective PredictionLanguage Model CalibrationConformal Risk Control
AskPDF
Oct 13, 2026

Adaptive Computation with Conformal Certification

Scientific imaging data involving medical (e.g., 3D radiographic, microscopic images) and astronomical sensors pose different challenges from natural images. One key challenge is that these datasets are often orders of magnitude larger than natural images. Efficient training and inference on such data is challenging due to the increased hardware requirements, often needing bespoke solutions. Moreover, the relevant objects such as tumors and astronomical phenomena are sparsely distributed with respect to the background information. Standard efficiency techniques like quantization and pruning apply uniform scaling of all regions, potentially compromising the precision/fidelity of predictions. Thus, efficient methods that reduce computational requirements without losing resolution for relevant parts of the data and with uncertainty estimates on predictions are essential. We present Conformal Adaptation with risk guarantees for uncertainty-aware sparse computations. Our formalism is general, spanning dense pixel-level classification and regression tasks where hierarchical models predict at low resolution, certify the signal using conformal risk control, and refine the certified predictions. We benchmark our method on multiple tasks such as 3D medical image segmentation, 2D galaxy flux prediction, and 1D calcium spike rate prediction, achieving 2.52.5--72×72\times wall-clock speedup across the various tasks, including on commodity CPUs.

Shehtab Zaman, Aneesh Rangnekar, Nancy Lee +1

Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, USA · Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, USA

Adaptive ComputationConformal Risk Control
AskPDF
Oct 13, 2026

CALM-5G: Event-Level Conformal Risk Control for Predictive Service-Assurance Alarms

Predictive service assurance converts forecasts of key performance indicators into alarms, and that conversion decides whether the assurance function is deployable. We present CALM-5G, an assurance layer that operates above any forecasting backbone and issues alarms with distribution-free, finite-sample guarantees. It calibrates on service-level violation events, scores violation onset directly, selects its calibration scheme out of sample, and reports the alarm duty cycle with every certified miss rate. Four corpora are evaluated separately, namely a production cellular network, commercial 5G traces, an open radio access network testbed and our 5G standalone testbed. Risk control on the regression-derived score yields duty cycles of 0.24 to 0.91. Onset targeting with cross-validated calibration reduces time-in-alarm by 56.3% to 89.7% on three corpora, with no significant change on the fourth. The realised miss rate stays within a two-standard-error tolerance on three corpora and within one standard error of it on the fourth. Under label shift, pooled batch calibration exceeds that tolerance structurally, by 10.9 standard errors on the production corpus. Online calibration restores compliance. Calibration schemes cannot be ranked on the split that fits their thresholds. Miss rate and duty cycle are certified jointly where within-entity dependence leaves sufficient calibration units.

Ömer Faruk İşcanlı, Yağmur Sabucu Sandal

Siemens A.¸S., Istanbul, Türkiye

Online CalibrationConformal Risk ControlModel CalibrationConformal PredictionWireless Communications
AskPDF
Oct 5, 2026

Valid Stopping in Adaptive Generator-Verifier Loops

Numerous agentic workflows are based on a generator-verifier loop: a generator proposes candidates, a cheap verifier scores them, and the workflow terminates when a proposal is verified as good enough. The verifier typically proxies a more costly ground-truth oracle, and as the generator searches adaptively against it, false acceptances may accumulate. Proposals can pass the proxy but fail under the costlier ground-truth check. We study when to stop these loops while controlling the false discovery rate of the accepted proposals. Our construction introduces tools of independent interest in distribution-free statistical testing and conformal risk control, including analysis of ee-values constructed through index betting and a novel conformal risk control procedure for non-monotone losses. We validate the approach in synthetic settings and on a protein-design benchmark.

Mahmoud Hegazy, Michael I. Jordan, Aymeric Dieuleveut

CMAP, École polytechnique, IP Paris, France · Inria Paris, France · EECS, UC Berkeley, USA

FDR ControlE-ValuesOptimal StoppingConformal Risk ControlSequential Hypothesis Testing
AskPDF
Oct 4, 2026

G-CARB: Graph-Localized Conformal Agent Risk Budget for Compositional Harm

Small language model (SLM) agents need safety controls that track consequences across tool calls with little monitoring overhead. A private read, for example, becomes a leak when a later action sends that data outside the system. We introduce CARB (Conformal Agent Risk Budget), which calibrates when to stop an agent using a ledger of harm incurred before stopping. Under exchangeable episodes, standard conformal risk control bounds this declared loss in expectation over calibration and a future episode. G-CARB selects scorer evidence along observable dependencies from private sources to outgoing actions. The ledger still covers the entire executed history, and computing the gate score requires no additional language-model inference. On AgentDojo replay with two 14B backbones, G-CARB roughly halves scorer-input records at intermediate risk budgets while improving autonomous task completion relative to full-prefix scoring; random context of the same size achieves similar gains. Controlled examples show how retaining the relevant dependency can further avoid stopping benign work.

Zijun Yu, Yu Gu, Vahid Partovi Nia +1

Department of Mathematics and Statistics McGill University Montréal, QC, Canada · Department of Mathematics and Industrial Engineering Polytechnique de Montréal Montréal, QC, Canada

AI Agent SafetyConformal Risk ControlInformation Flow ControlLLM Agent Safety
AskPDF
Sep 29, 2026

ReCIRC: Rectified Conformal Risk Control

Many applications of black-box predictive models require controlling task-relevant error rates, such as missed lesion pixels in segmentation or missed labels in multilabel classification. Conformal risk control (CRC; Angelopoulos et al., arXiv:2208.02814) gives distribution-free guarantees for such losses, but it calibrates a single threshold shared by all inputs. Because conditional risk varies with the input, this marginal guarantee often overprotects easy cases and underprotects hard ones. We propose ReCIRC (Rectified Conformal Risk Control), which inverts each input's estimated local risk curve to reparameterize the calibrated threshold as a risk budget aa representing a common target conditional risk, and then applies CRC unchanged to the resulting family. ReCIRC retains CRC's finite-sample marginal guarantee regardless of the accuracy of the estimated curves, while accurate curves yield approximate conditional risk control and, under additional conditions, asymptotically exact conditional risk control; they also support a risk-calibration diagnostic. Across three synthetic and five real-data settings spanning segmentation, multilabel and multiclass classification, and regression, ReCIRC attained the lowest average worst-group risk and mean positive group excess in every setting, while maintaining marginal risk close to the target, whereas changes in prediction size were application-dependent.

Bruno Marcondes e Resende, Helton Graziadei, Thiago Rodrigo Ramos +1

Department of Statistics, Federal University of São Carlos

Conformal PredictionConformal Risk Control
AskPDF
Sep 28, 2026

Adaptive Safety Filtering for Frozen ACC Policies via Conformal Residual Calibration

Frozen adaptive cruise control (ACC) policies can violate constraints when deployment dynamics differ from their training conditions. We propose residual-aware conformal action filtering (RACF), which calibrates residuals of a fixed nominal predictor and converts their quantile into an operating margin for finite-model action projection. Completed transitions update margins and candidate selection without retraining the policy. In a registered comparison over 2,400 controller-trial units, Adaptive RACF achieves 94.3% episode safety, improving by 19.9 percentage points over the evaluated nominal CBF-QP baseline while reducing projection frequency from 8.11% to 6.63%. A controlled study isolates a 4.54-point improvement from residual-margin injection. In a separate matched-hardware evaluation, Adaptive reduces mean amortized rollout time by 21.2% relative to Robust CBF-QP, with 161/180 versus 170/180 safe episodes. We characterize conditions linking one-step residual coverage to constraint satisfaction and quantify the observed safety-computation trade-offs.

Zhiruo Zhou, Rigaudiere Z. Li, Chen Xiwen +3

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Wuhan University of Technology, Wuhan, China · Shanghai Jiao Tong University, Shanghai, China

Safety-Critical ControlAutonomous Driving Safety EvaluationAutonomous DrivingSafety FilteringConformal Risk Control
AskPDF
Sep 27, 2026

Faster Block-Diffusion Serving with Distribution-Free Risk Guarantees

Block-diffusion language models are served at hand-picked operating points, such as acceptance thresholds, buffer depth, schedule, checkpoint and precision, and each point is chosen by its mean benchmark accuracy. However, a mean does not tell an operator how often a faster configuration fails on prompts that the slower one answers correctly. On the serving engine and its decode traces, the default commit rule already commits every fully resolved block, a static skip rule captures nearly all of the compute that allocation can save, and self-distillation on engine-decoded targets adds speed at unchanged accuracy. Larger speedups come from lower thresholds, which commit tokens that are still uncertain. We therefore present Redline, a finite-sample procedure that selects operating points, hand-picked or learned, from the correctness of their answers on calibration prompts. Redline keeps the reference-relative risk, the joint probability that the reference answers correctly and a candidate configuration does not, within a user-chosen budget with high probability, and deploys the fastest configuration that passes. It speeds up math at a smaller risk budget than code in both model families, and at a budget of ten percent it deploys a LLaDA2 math configuration that commits over a third more tokens in each forward. It also applies without modification to the acceptance rule of speculative decoding and to weight quantization. On the same calibration data, Redline stays within its stated failure probability, whereas each tolerance of a mean-accuracy rule either gains less speed for some model and task or exceeds the risk budget far more often for another. Code is available at https://github.com/js-lee-AI/Redline.

Jungseob Lee, Dongyub Jude Lee, Chanjun Park +2

Korea University · Zoom Communications · Soongsil University +1

Diffusion Language Model InferenceLLM Inference AccelerationConformal Risk ControlDiffusion Language ModelsDiffusion Model Inference Acceleration
AskPDF
Sep 22, 2026

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

A conformal abstention certificate for text-to-SQL is only as truthful as the correctness labels it is calibrated on. The uncertainty pipelines that read confidence off execution consistency take those labels from the single database a benchmark ships, an oracle known to be lenient. We run a preregistered intervention on Spider-Realistic, swapping that database for the benchmark's distilled multi-instance test suite. Across four SQL-specialist checkpoints and two split schemes, the swap raises the certificate's held-out risk 2.73 to 10.23 points above the risk its own labels report. Neither oracle reports the risk experts assign. Under blinded labels from two SQL experts, a certificate calibrated at a nominal 0.10 carries 20.0 and 17.2 points of risk on two checkpoints. The stricter oracle errs in both directions: most of the answers it rejects are not judged wrong, and some of those it accepts are. An AI-assigned census of what it rejects finds a semantic error in a quarter to a third of them, depending on the population. It attributes most of the rest to underspecified questions, synthetic instances or suspected reference-query defects, a flag supported by a preregistered blinded expert audit. The oracle also decides how a confidence score is judged. Every execution-consistency score looks better under the labels of the oracle that built its clusters, in 16 of 16 combinations. Under expert labels, building such a score on suite clusters instead of shipped-database clusters raises its area under the ROC curve (AUROC) by 6.96 points on one checkpoint and 1.53 on the other. On the second, the expert interval excludes the 8.3 points the suite labels report. A certificate should be reported with both oracles, and an oracle-relative difference read as semantic risk only after the benchmark is audited. A consistency score should be evaluated under an oracle that did not build it.

Jiamiao Liu, Dewen Qiao, Yu Zhang +1

Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China

Probability CalibrationBenchmark AuditingConformal PredictionText-to-SQLConformal Risk ControlBenchmark Validity
AskPDF
Sep 2, 2026

Occupancy-based Quantile Risk Control

Conformal risk control is an emerging framework for the safe deployment of machine learning models with finite-sample guarantees. To accommodate a broader class of risk notions, quantile risk control extends this framework to quantile-based risk measures. However, existing methods either suffer from excessive conservatism or lack rigorous finite-sample guarantees. To address these limitations, we introduce Occupancy-based Quantile Risk Control (OQRC), a novel method that provides tight risk control bounds with finite-sample validity. Our key idea is to formulate risk control as a finite-occupancy problem by partitioning the loss space with the ordered calibration losses. Specifically, we estimate the distribution of test losses across the resulting bins and upper-bound the risk by the maximum loss attained within each bin. We then select the parameter λλ such that this upper bound does not exceed a predefined threshold αα with high probability 1−δ1-δ. Theoretically, we establish a finite-sample guarantee showing that OQRC yields tight risk control bounds that converge to the optimal bounds at a provable rate of Op(n−1/2)\mathcal{O}_ p(n^{-1/2}). Extensive experiments demonstrate the effectiveness of our method, reducing the risk gap by up to 78.64% on common benchmarks.

Zihao Shi, Huajun Xi, Bingyi Jing +1

Southern University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong, Shenzhen +1

Conformal Risk ControlStatistical Learning Theory
AskPDF
Aug 31, 2026

Safety Screening for Voltage Control in Active Distribution Grids via Distributionally Robust Conformal Screening

Deploying a new control policy for voltage control in active distribution grids requires evidence that physical limits will be satisfied before the policy is tested on the physical grid. This assessment is difficult for two reasons. First, simulations cannot capture every disturbance, modeling error, and device interaction present in the real grid. Second, historical measurements reflect operation under existing control policies, whereas a new policy may drive the grid into different operating conditions. To address these challenges, we propose Distributionally Robust Conformal Safety Screening (DR-CSS), a policy-agnostic framework for pre-deployment, scenario-by-scenario screening of a new control policy using historical data and a nominal simulator. For each new scenario, the simulator predicts a future voltage trajectory for the whole grid; DR-CSS then constructs a conformal safety interval around this prediction using historical simulation-to-reality errors. The interval is further enlarged to account for closed-loop changes induced by the deployment of the new policy and its interactions with the remaining controllers. To the best of our knowledge, DR-CSS is the first framework in power systems to combine historical data from an existing control policy with an imperfect simulator for pre-deployment safety screening of a new policy. Experiments on the IEEE 33-bus and IEEE 141-bus systems evaluate the deployment of learning-based voltage control policies and show that DR-CSS identifies all unsafe test scenarios. To reduce unnecessary warnings on safe scenarios, we adapt the safety intervals to different operating conditions and gradually introduce new policies with recalibration after each stage. These extensions increase the informational value of the safety screening and support safer deployment decisions in active distribution grids.

Sarra Bouchkati, Petros Ellinas, Adriana Geisler +4

IAEW at RWTH Aachen University, Aachen, Germany · Department of Wind and Energy Systems, Technical University of Denmark, Kongens Lyngby, Denmark

Safety-Critical ControlConformal Risk ControlPower Systems
AskPDF
Aug 28, 2026

Conformal Risk-Averse Decision Making with Optimized Certainty Equivalent Risk Control

We study risk-averse decision making, in which an agent selects actions while being uncertain about the true system state. The risk is measured via optimized certainty equivalent (OCE) metrics, which generalize popular criteria such as mean-variance risk and conditional value-at-risk (CVaR). We characterize the optimal policy under known distributions, and show that it reduces to a prediction set-based solution for the CVaR. This provides an operational interpretation of conformal prediction-type prediction sets. For unknown distributions, we develop a data-driven calibration strategy, based on a synthetic model for the likelihood and held-out calibration data, yielding high-probability control of the OCE risk. The approach is evaluated on two wireless beamforming settings.

Amirmohammad Farzaneh, Osvaldo Simeone

Institute for Intelligent Networked Systems (INSI), Northeastern University London, London, UK

Risky Decision-MakingDecision-Making under UncertaintyConformal Risk ControlConditional Value-at-Risk
AskPDF
Aug 27, 2026

TRACE-CRC: Trajectory-Adaptive Conformal Risk Control for Multi-Step Channel State Information Prediction

Reliable prediction of time-varying channel state information (CSI) is essential for efficient wireless communication. Each CSI frame is a matrix-valued representation of the wireless channel response, and a sequence of CSI frames forms a temporal channel trajectory. Modern deep learning-based CSI predictors, however, often provide only point predictions and lack calibrated uncertainty estimates. This limitation is particularly problematic in multi-step CSI prediction, where the target is a sequence of future CSI matrices, and downstream decisions such as beamforming or scheduling may fail if any part of the predicted trajectory is unreliable. We propose trajectory-adaptive calibration and error profiling with conformal risk control (TRACE-CRC), a method for trajectory-aware uncertainty quantification in multi-step CSI prediction. TRACE-CRC constructs Frobenius-norm uncertainty balls around predicted CSI matrices and controls the risk that at least one future frame is uncovered. Instead of calibrating each future step independently, TRACE-CRC combines future-step-dependent error profiling, trajectory difficulty stratification, and learn-then-test (LTT) risk control. Empirically, TRACE-CRC achieves reliable trajectory-level coverage with substantially smaller uncertainty balls than conservative multi-step corrections, while avoiding the trajectory undercoverage of compact stepwise and adaptive conformal baselines.

Kiarash Rezaei, Mehdi Sattari, Javad Aliakbari +3

Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden

Channel State PredictionUncertainty QuantificationConformal PredictionConformal Risk ControlTrajectory Prediction
AskPDF
Aug 27, 2026

Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue

In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts' distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC3): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function Ct(x)C_t(x) applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC3 extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.

Ming Cheng, Yusheng Dai, Qiuhong Ke +2

Monash University

LLM-as-a-JudgeConformal Risk ControlPairwise Preference EvaluationMulti-Turn Dialogue Evaluation
AskPDF
Aug 18, 2026

Bound-Aware Per-Organ Recall Risk Control for Multi-Organ CT Segmentation under Clinical Domain Shift

Distribution-free risk control adds organ-specific recall guarantees to frozen segmentation. We calibrate per-organ thresholds for an AMOS-trained nnU-Net, audit transfer to RAOS, and estimate local re-certification cost using case-level voxel false-negative rate (FNR). The AMOS control passes, but 7/127/12 organs exceed α=0.10α{=}0.10 after transfer; smaller calibration sets can mask exceedances with conservative or vacuous thresholds. Risk-Controlling Prediction Sets (RCPS) give high-probability control of population-mean risk, whereas Conformal Risk Control (CRC) gives weaker expectation control. Both require exchangeability; fixed and global thresholds give no per-organ guarantee. The Waudby--Smith--Ramdas (WSR) betting bound re-certifies six Tier-1 organs with 25 local cases, versus 30--40 for Hoeffding--Bentkus (HB). CRC needs 10--15 but has a heavier individual-case tail. No Tier-2 organ meets our illustrative precision criterion with 25 cases.

Souraj Adhikary, Negar Chabi, Andre Mastmeyer

Department of Engineering Sciences, Jade Hochschule Wilhelmshaven, Germany

Multi-Organ SegmentationMedical Image SegmentationMedical Image AnalysisConformal Risk ControlComputed Tomography
AskPDF
Aug 14, 2026

TRIAGE: Risk-Controlled Pseudo-Label Admission for Annotation-Efficient Semi-Supervised Retinal OCT Classification

The advanced retinal disease diagnosing imaging modality, optical coherence tomography (OCT), encounters a lack of automation because of the high expenses for annotations performed by specialists. The use of SSL solves the problem of insufficient annotations using unlabeled B-scans; however, most of the current techniques for generating pseudo-labels are based on prediction confidence without considering the asymmetry between different types of errors. This paper proposes TRIAGE, a risk-controlled semi-supervised framework for OCT scans classification, which uses the concept of a patient-level conformal risk controller with an asymmetric cost matrix. TRIAGE unites three crucial modules: a hierarchical classifier that is capable of working with partially abnormal supervision of the disease subtypes, a patient-grouped conformal risk controller with primal-dual coverage control, and a context-aware Transformer teacher for cross-slice verification. On the dataset from Noor Eye Hospital (16,822 B-scans, 161 patients, and 554 volumes) with a test set of unseen patients, TRIAGE demonstrates 89.66% scan-level accuracy, 0.8805 macro-F1, 0.9641 macro-AUC, and an 8.34% under-grading rate when using only 20% of the labeled data. With only 5% of the labeled data, TRIAGE keeps 76.88% accuracy and a 0.1656 under-grading rate. Compared with the other six state-of-the-art semi-supervised methods, TRIAGE significantly outperforms them with ablation study demonstrating the contribution of each module in the overall framework performance (by 42.7% in terms of under-grading rate comparing to fixed threshold methods). TRIAGE demonstrates 98.00% accuracy for 3-class classification with 1% labeled data and 95.94% accuracy for 8-class classification with 10% labeled data on the OCT-C8 dataset.

Md Ashraful Hossen Akash, Shyla Afroge, Abdullah Al Mamun +2

Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi-6204, Bangladesh · Elite Research Lab LLC, New York, USA · Faculty of Information Science and Technology, Multimedia University, Melaka, Malaysia +1

Medical Image ClassificationRetinal Disease ClassificationPseudo-LabelingOptical Coherence TomographyConformal Risk Control
AskPDF
Aug 4, 2026

Conformal risk control for model-form uncertainty in parametric non-intrusive reduced-order models

Non-intrusive reduced-order models (NIROMs) have become a standard tool for approximating parametric partial differential equations from computer design of experiments while significantly reducing computational costs. However, assessing the reliability of their predictions remains a major challenge, particularly in extrapolation regimes or under limited training data. In this work, we introduce a framework for quantifying model-form uncertainty in NIROMs by combining a perturbative stochastic representation of reduced bases with distribution-free conformal-type methods. Starting from a deterministic reduced basis constructed from snapshot matrices, we model uncertainty through random perturbations defined on the Stiefel manifold, directed along the discarded modes, yielding stochastic reduced-order approximations whose induced variance reflects the basis-truncation error. A transport approximation gives a closed-form posterior variance that separates basis-induced from regression-induced uncertainty, without re-training the underlying Gaussian processes. We include this posterior variance within a conformal risk control calibration framework, that provides prediction sets with coordinate miscoverage guarantees. The calibration factor produced by this framework is itself an interpretable, scalar diagnostic of the quality of the uncertainty estimate. The methodology is evaluated on parametric PDE benchmarks and an industrial tire-manufacturing calendering process. Numerical experiments demonstrate reliable, locally informative uncertainty quantification that goes beyond the Gaussian predictive variance.

Edgar Jaber, Rémy Vallot, Thibault Dairay +1

CB, ENS Paris Saclay · Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, Gif-sur-Yvette, 91190, France · CB, Michelin +3

Uncertainty QuantificationPDE Surrogate ModelingConformal Risk ControlReduced-Order Modeling
AskPDF
Jul 27, 2026

Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models

Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style. Conformal risk control (CRC) gives rigorous marginal risk guarantees for selective prediction with abstention, but marginal guarantees do not imply per-group ones: a model can meet the population budget while systematically over-exposing subgroups to errors. Under mild shift in group composition, standard CRC violates the budget in up to 47% of trials. We propose HG-CRC (Hierarchical Group-Conditional CRC), a post-hoc calibration framework enforcing simultaneous risk guarantees across all nodes of a user-defined group hierarchy. It applies a Bonferroni correction over nodes and a leaf-first policy that uses the most specific applicable threshold, falling back to coarser nodes when a finer one is uncertified or rejects the example. It needs only a held-out calibration set, with no retraining. We evaluate on three models (Qwen3-4B, Llama-3.1-8B-Instruct, Gemma-3-4B) and two benchmarks (ARC Challenge, MMLU-Pro) across eight configurations probing IID generalization, heterogeneity, mixture/domain/prompt/difficulty shift, label noise, and quantization. Main result: HG-CRC reaches an empirical 0% violation rate and WGER=0 on ARC Challenge for high-accuracy models (Qwen3-4B, Llama-3.1-8B). At 500 bootstrap trials these zeros are empirical upper bounds (true rate up to 0.6%), not certified. Results are benchmark-specific: on MMLU-Pro these models abstain entirely or (Llama) retain WGER=0.014. Gemma-3-4B, poorly calibrated here, degrades gracefully by abstaining. Participation cost vs. global CRC is 22 to 37 points. Ablations show hierarchical depth clears the budget: removing difficulty level returns violations to about 11%. Bonferroni is needed for the theoretical guarantee, though its empirical effect matters only with many nodes.

Murilo Salem, Luísa Böhm, Daniel Pontes +1

CDTec (Centro de Desenvolvimento Tecnológico) Universidade Federal de Pelotas Pelotas, RS, Brasil

Selective PredictionGroup-Conditional Conformal PredictionLanguage Model CalibrationConformal Risk ControlLanguage Model Robustness
AskPDF
Jul 27, 2026

Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls

Language-model agents act through structured tool calls whose arguments carry very different risks: untrusted content may legitimately shape an email body but should never set a recipient, account, command, or credential. Existing conformal risk control methods certify a tool call as a whole, so a failure in one rare high-risk field can be averaged away by the many benign arguments around it, leaving the argument that causes harm uncertified. We introduce role-stratified per-field conformal risk control, a calibration layer that wraps any per-field detector and assigns a separate threshold and risk budget to each semantic argument role. We show that aggregate certification pays a price of coarseness, tightening a rare role's effective budget in proportion to how often that role appears, whereas role-stratified calibration certifies each sufficiently sampled role directly with a finite-sample guarantee and pools the rarest roles. Across AgentDojo and InjecAgent with six language models, our method achieves the most consistent role-specific budget compliance among the methods we evaluate under model and attack transfer, detector noise, gradual drift, unseen tool suites, and adaptive attacks, providing formal per-role guarantees under exchangeability or after recalibration. These results suggest that structured tool calls should be certified at the semantic-role level, not the whole action.

Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin +4

Conformal Risk ControlLLM Agent Safety
AskPDF
Jun 27, 2026

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.5--12.5%. We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot. First, we prove an impossibility result: when the base risk (μ> α), any distribution-free method must abstain on at least ((μ-α)/(1-α)) examples, yielding a closed-form feasibility test: one can check whether CRC will work before running it. Second, we analyze a certification hierarchy across Hoeffding, empirical Bernstein, and a betting-based e-CRC bound, with strict gains in low-variance/large-sample regimes: the Hoeffding-to-Bernstein step delivers the largest gain (+37% certified configurations), while e-CRC adds value when calibration data is scarce (10% certification at 20% data versus 0% for Hoeffding). Third, we validate adaptive conformal inference (ACI) under cross-dataset shift, reducing risk-target violations from 71% to 21%, with residual failures concentrated exactly where the impossibility bound predicts. Across six open-weight models (3B--72B parameters), eight datasets, four tasks, and six nonconformity scores, hard NER/QA/CLS configurations are uncertifiable at (α= 0.10); relaxing to (α= 0.30--0.40) unlocks practical certification (47% NER, 40% QA, 60% CLS). The framework gives a three-step deployment recipe: check feasibility, select the bound and score, then mitigate shift.

Varun Kotte

Independent Researcher

Conformal PredictionLLM ReliabilityConformal Risk ControlAdaptive Conformal InferenceStructured Output GenerationGeneralization Bounds
AskPDF
Jun 24, 2026

Conformal Recovery-Deadline Certificates for Runtime Assurance of Adapting Controllers

Runtime assurance (RTA) protects a safety-critical system by switching from an advanced controller to a verified safe controller when a monitored condition is violated. The standard latching rule, which trips on the first breach of the safe set and then coasts, is correct for a diverging controller but pathological for a capable online-adapting one. Such a controller is unsafe by design during a bounded recovery transient. It must excite the plant to identify the fault before it can correct it, so a latching shield trips on that transient and suppresses a controller that would have recovered. We introduce the conformal recovery-deadline certificate, a split-conformal, distribution-free, finite-sample upper bound on the adapting controller's recovery time that licenses delayed fallback with a coverage guarantee, backstopped by a verified monitor at a hard critical limit. The certified deadline discriminates capable from incapable controllers, keeping the recoverer autonomous while catching the diverger. The construction separates autonomy, governed by statistical coverage, from safety, governed by the verified backstop, as an instance of reliability-asymmetric design. We prove marginal coverage, a weighted extension that restores coverage under a known fault-distribution shift, and group-conditional Mondrian coverage. We demonstrate all three on two unrelated Simplex testbeds: a 6-DOF spacecraft attitude controller and a torque-controlled inverted pendulum. Both show the same suppression pathology and the same cure, making the certificate a domain-general mechanism rather than a single-system trick.

Alireza Shojaei

Myers-Lawson School of Construction, Virginia Tech, Blacksburg, VA 24061 USA

Safety-Critical ControlConformal PredictionConformal Risk ControlRuntime AssuranceAdaptive Control
AskPDF
Jun 18, 2026

When Calibration Fails the Vulnerable Hospital: Federated Conformal Risk Control via Risk-Curve Shrinkage

Conformal risk control (CRC) provides distribution-free guarantees on segmentation quality by calibrating a prediction-set threshold on held-out data. In federated deployments, the standard approach pools calibration scores across sites into a single threshold. We provide the first quantification, on real multi-institutional brain tumor data (FeTS-2022, 1,251 subjects, 20 institutions), showing that this naive pooled CRC protects the average hospital but violates coverage at 40% of individual institutions, with the worst site exceeding the target false-negative rate by 7.8 percentage points. The naive alternative, per-site local CRC, largely restores coverage but inflates prediction sets by 83x, rendering them clinically useless. We propose a shrinkage-based federated CRC protocol: each site transmits only its empirical risk curve (G scalars) to a server, which computes a shrinkage-regularized threshold per site. A single hyperparameter n0 smoothly trades worst-case coverage for prediction-set efficiency; leave-one-site-out sensitivity analysis identifies n0=19, achieving 2.7/20 violations at 2.0x stretch. We further show that direct Lagrangian optimization of coverage budgets fails, concentrating risk on vulnerable hospitals, and that the finite-sample correction term is essential: removing it triples violations. The marginal CRC guarantee is preserved by construction under the stated site-mixture assumption; per-site coverage is validated across four targets with three seeds. No patient-level images, masks, or per-volume scores leave any site.

Nafis Fuad Shahid

Dhaka, Bangladesh

Federated Conformal PredictionConformal PredictionConformal Risk ControlFederated Learning
AskPDF
Jun 16, 2026

ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift

Modern AI agents retrieve documents, call tools, check intermediate information, and then produce a final answer or action. This creates a risk-control problem that is not visible from the final answer alone. A final response may look acceptable even when the retrieval was weak, a tool output was wrong, or an earlier step was unsupported. We propose ToolChain-CRC, a conformal risk-control method for retrieval-augmented and tool-using agents under drift. The method treats each agent run as a full trajectory of actions, observations, and final output. It builds step-level risk scores, combines them into a trajectory risk score, calibrates an accept-or-intervene rule, and adds an anytime alarm that can stop risky runs before the final answer. We prove trajectory-level risk control under exchangeable calibration runs, give a drift-aware extension with auditable constants, and prove an anytime escalation rule through a supermartingale construction. Experiments cover synthetic tool-chain drift, RAG/tool-use stress tests, public SQuAD-derived retrieval tasks, an API-free agentic QA case study, ablations, target-risk sensitivity checks, 20-seed robustness checks, a drift-margin audit, and a live RAG/tool-use agent benchmark. Across these settings, final-answer-only calibration can miss retrieval and tool failures, while trajectory-level calibration keeps accepted-trajectory risk below the target.

Jeffery Opoku, David Banahene

The University of Texas Rio Grande Valley, Edinburg, TX, USA · Florida International University, Miami, FL, USA

AI Agent ReliabilityDistribution ShiftAI ControlTool-Using AgentsConformal Risk ControlAgentic RAG+1
AskPDF