Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection

Jing Li; Haibin Zhu

stat.MLSep 22, 2026

Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection

Authors: Jing Li, Haibin Zhu

Organizations: Department of Statistics and Data Science, School of Economics, Jinan University, Guangzhou, China.

Abstract

Modern conformal forecasting systems often maintain several adaptive pipelines that differ in base forecasters, conformity scores, calibration windows, and update rules. Comparing them is difficult because coverage is a hard constraint, whereas efficiency should be optimized only among feasible pipelines. We formulate this problem as sequential inference for a stochastic constrained argmin. At each time, the target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints. We introduce Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS), which separate certifiably feasible, possibly feasible, and possibly constrained-optimal pipelines. Using simultaneous martingale confidence sequences, CC-SMCS projects a rectangular confidence region onto the constrained argmin and admits an exact closed-form rule. With probability at least $1-δ$ , it contains every constrained-optimal pipeline simultaneously over all times. This finite-sample guarantee requires no stationarity or mixing assumptions and remains valid under data-dependent stopping. We also establish an impossibility result for safe certification at the coverage boundary and extend the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.

Figures & tables

Simulations	$\delta$	$P_{TU}$	$\bar{\mathrm{rad}}_{t_{0.25}}$	$\bar{\mathrm{rad}}_{t_{0.50}}$	$\bar{\mathrm{rad}}_{t_{0.75}}$	$\widehat{p}_{\tau,T}$
Simulation 1	0.10	1.000	0.0361	0.0234	0.0186	0.001
Simulation 1	0.25	1.000	0.0326	0.0211	0.0168	0.001
Simulation 2	0.10	1.000	0.0402	0.0283	0.0231	–
Simulation 2	0.25	1.000	0.0363	0.0255	0.0209	–
Simulation 3	0.10	0.999	0.0372	0.0252	0.0209	–
Simulation 3	0.25	0.999	0.0336	0.0227	0.0188	–

Table 1: Simulation results

Panel of Figure 1 — Figure 1: The performance of simulations with $\delta=0.10$ . The vertical dashed line in Figure 1(b) indicates the structural break.

Panel of Figure 2 — Figure 2: Empirical results for DLinear at 95% nominal coverage with $\delta=0.10$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$C_{mrt}$	Prediction set from method $m$ under constraint $r$ at round $t$
$B_{mrt}$	Miscoverage indicator $\mathbf{1}\{Y_{t}\notin C_{mrt}\}$
$a_{rt}$	Predictable exposure or subgroup weight
$\Gamma_{mr,t}$	Prefix-average conditional excess miscoverage
$Q_{m,t}$	Observed prefix-average set cost
$\mathcal{V}_{t}^{\star}$	True feasible method set

Appendix

Table 2: Main notation.

The selection of $\rho$	$\delta$	Simulation Data Set	Data Set 1	Data Set 2	Data Set 3	Data Set 4
$\rho^{\star}$	0.10	17.616	20.494	29.584	30.946	8.151
$\rho^{\star}$	0.25	21.752	25.305	36.529	38.211	10.064

Appendix

Table 3: Selected values of

\rho

Panel of Figure 3 — Figure 3: The performance of simulations with $\delta=0.25$ . The vertical dashed line in Figure 3(b) indicates the structural break.

Panel of Figure 4 — Figure 4: Empirical results for DLinear at 95% nominal coverage with $\delta=0.25$ .

Panel of Figure 5 — Figure 5: Empirical results for PatchTST at 95% nominal coverage with $\delta=0.10$ .

Panel of Figure 6 — Figure 6: Empirical results for PatchTST at 95% nominal coverage with $\delta=0.25$ .

Panel of Figure 7 — Figure 7: Empirical results for TimesNet at 95% nominal coverage with $\delta=0.10$ .

Panel of Figure 8 — Figure 8: Empirical results for TimesNet at 95% nominal coverage with $\delta=0.25$ .

Explore similar work

Simultaneous Coverage and Efficiency Guarantee in Online Conformal Prediction
Jul 29, 2026Rahul VazeSet Prediction Conformal Prediction
Tata Institute of Fundamental Research, Mumbai
Adaptive Conformal Inference through the Lens of Blackwell Approachability
Oct 17, 2025Guillaume Principato, Gilles StoltzConformal Prediction Adaptive Conformal Inference
Universit´e Paris-Saclay, CNRS, Inria, Laboratoire de math´ematiques d’Orsay, 91405, Orsay, France · EDF R&D, Palaiseau, France · HEC Paris, Jouy-en-Josas, France
Online Conformal Prediction: Enforcing monotonicity via Online Optimization
May 12, 2026Eduardo Ochoa Rivera, Ambuj TewariConformal Prediction Uncertainty Calibration
University of Michigan · Ann Arbor, USA

Jul 29, 2026J/K move · Enter open · S save

Simultaneous Coverage and Efficiency Guarantee in Online Conformal Prediction

Rahul Vaze

Tata Institute of Fundamental Research, Mumbai

Set Prediction Conformal Prediction Adaptive Conformal Inference Statistical Learning Theory

Open paper AskPDF arXiv

Adaptive conformal inference (ACI) of Gibbs and Cand{è}s and its variants are the standard approach to online conformal prediction under distribution shift, but they suffer from three fundamental limitations. First, their guarantees control only the signed long-run coverage error: persistent miscoverage in one direction can be masked by compensating errors later, so a method can satisfy the theoretical guarantee while being badly wrong for extended periods. Second, existing guarantees say nothing about prediction-set size, so validity can be achieved trivially at the cost of unduly wide prediction sets. Third, the efficiency guarantees that do exist compare against a fixed predictor chosen in hindsight, a benchmark that becomes increasingly less meaningful once the data-generating distribution shifts, since the very notion of an optimal threshold then changes over time. We consider a unified online learning framework that simultaneously controls absolute, non-cancelling coverage violation and prediction-set efficiency against a dynamically evolving benchmark for three important models. In the fully adversarial setting, exploiting the fact that the standard ACI update is exactly projected online gradient descent on the pinball loss, we derive simultaneous coverage and efficiency guarantees for arbitrary monotone Lipschitz efficiency objectives, with no distributional or {\it convexity} assumptions. In the stochastic setting with full-score feedback, we propose a sliding-window quantile tracker and establish a matching minimax lower bound showing our algorithm is rate-optimal. In the covariate-dependent stochastic setting, we develop a partitioned ACI algorithm that tracks a function-valued oracle threshold, and derive simultaneous coverage and efficiency guarantees.

Abstract

Explore similar work

Simultaneous Coverage and Efficiency Guarantee in Online Conformal Prediction

Adaptive Conformal Inference through the Lens of Blackwell Approachability

Online Conformal Prediction: Enforcing monotonicity via Online Optimization