Counterfactual Online Conformal Prediction Under Adaptive Logging

Xinyu Qiao; Yichen Lin; Kaihong Ji; Xue Wang; Tao Yao

cs.LGSep 25, 2026

Counterfactual Online Conformal Prediction Under Adaptive Logging

Authors: Xinyu Qiao, Yichen Lin, Kaihong Ji, Xue Wang, Tao Yao

Organizations: Shanghai Jiao Tong University, Shanghai, China · Alibaba Group

Abstract

Online conformal prediction can fail when predictions shape actions and actions determine which outcomes enter calibration. Standard adaptive methods may retain marginal coverage while systematically miscovering the counterfactual outcomes of rarely selected actions. This paper formalizes the failure through counterfactual coverage and introduces Propensity-Weighted Online Conformal Prediction, an inverse-propensity-weighted recursion that debiases calibration. A doubly robust variant further reduces nuisance bias to the product of outcome-model and propensity errors. Under positivity, the resulting coverage rate matches an information-theoretic lower bound up to logarithmic factors. Experiments on synthetic decision tasks, open bandit data, and financial rebalancing show that PW-OCP and DR-OCP improve counterfactual coverage and downstream regret without sacrificing prediction-set sharpness.

Figures & tables

Panel of Figure 1 — Figure 1: Logged coverage can hide counterfactual failure. Both panels sweep the mean-shift strength $\Delta\in[0,3]$ at $T=20{,}000$ and $\alpha=0.1$ . The left panel shows that ACI’s logged marginal gap stays essentially zero on a $10^{-3}$ scale. The right panel shows the counterfactual gap: ACI grows with $\Delta$ , while PW-OCP remains close to Oracle, reproducing and correcting the $\Omega(\rho)$ failure in Theorem 3.2 . Bands are $95\%$ confidence intervals over $10$ seeds.

Panel of Figure 2 — Figure 2: Counterfactual coverage gap across three synthetic stress tests. Mean-shift and variance-shift simulations sweep endogeneity at $K=2$ : in the first panel $\Delta$ is a mean shift, and in the second it is a standard-deviation increment. The multi-arm panel sweeps $K\in\{2,5,10,20,50\}$ . Exogenous baselines retain persistent gaps, whereas PW-OCP and DR-OCP track Oracle at the rate predicted by Theorem 5.1 . Bands are $95\%$ confidence intervals over $10$ seeds.

Method	Mean shift	Variance shift	Multi-arm	Propensity error	OBD Women	OBD Men	DJIA	Regret vs ACI
ACI [ 9 ]	0.252 $\pm$ 0.002	0.230 $\pm$ 0.002	0.418 $\pm$ 0.009	0.252 $\pm$ 0.004	0.023 $\pm$ 0.005	0.025 $\pm$ 0.005	0.024 $\pm$ 0.003	—
FACI [ 10 ]	0.275 $\pm$ 0.001	0.250 $\pm$ 0.002	0.503 $\pm$ 0.009	0.275 $\pm$ 0.004	0.041 $\pm$ 0.006	0.043 $\pm$ 0.005	0.055 $\pm$ 0.007	-7.0% $\pm$ 1.1%
AgACI [ 36 ]	0.250 $\pm$ 0.001	0.230 $\pm$ 0.002	0.411 $\pm$ 0.007	0.251 $\pm$ 0.004	0.027 $\pm$ 0.005	0.033 $\pm$ 0.008	0.024 $\pm$ 0.003	+0.5% $\pm$ 1.1%
Conformal PID [ 1 ]	0.290 $\pm$ 0.002	0.261 $\pm$ 0.002	0.689 $\pm$ 0.003	0.251 $\pm$ 0.004	0.245 $\pm$ 0.005	0.262 $\pm$ 0.011	0.024 $\pm$ 0.004	-20.5% $\pm$ 1.2%
SAOCP [ 5 ]	0.269 $\pm$ 0.004	0.245 $\pm$ 0.002	0.450 $\pm$ 0.008	0.270 $\pm$ 0.005	0.042 $\pm$ 0.003	0.047 $\pm$ 0.006	0.074 $\pm$ 0.004	-4.5% $\pm$ 1.4%
NEx-CP [ 4 ]	0.255 $\pm$ 0.002	0.234 $\pm$ 0.002	0.422 $\pm$ 0.007	0.257 $\pm$ 0.003	0.025 $\pm$ 0.005	0.028 $\pm$ 0.005	0.024 $\pm$ 0.007	-1.0% $\pm$ 1.1%

Table 1: Master results across all benchmarks. Columns report counterfactual coverage gaps (lower is better) and regret reduction on the robust decision rule of Theorem C.5 , normalized so ACI is

0\%

and Oracle is

100\%

(higher is better). Entries are means

\pm 95\%

confidence intervals over

10

seeds. Red and blue mark the best and second-best non-oracle methods.

Panel of Figure 3 — Figure 3: Robustness, rates, and prediction-set quality. Left : propensity misspecification is induced by giving the learner $\hat{\pi}=(1-\xi)\pi+\xi/K$ , where $\xi=0$ is the true logger and $\xi=1$ is uniform. PW-OCP degrades as the IPW correction loses the true propensity, while DR-OCP remains stable because the outcome model absorbs the first-order propensity bias predicted by Theorem 5.2 . Middle : as the horizon $T$ grows, PW-OCP and DR-OCP drive the counterfactual coverage gap toward Oracle, whereas logged-calibration baselines retain persistent bias. Right : the average finite threshold error $\bar{\eta}_{T}$ relative to the oracle counterfactual quantile decreases with $T$ , showing that the methods improve coverage without simply widening prediction sets and matching the sharpness behavior in Theorem C.3 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Logged coverage diagnostics across synthetic settings. The top row reports marginal coverage $\mathrm{MCov}_{T}$ and the bottom row reports action-conditional coverage gaps for the same mean-shift, variance-shift, and multi-arm settings as Figure 2 . These logged diagnostics can appear well calibrated even when the corresponding counterfactual gaps remain large, illustrating why logged metrics alone are insufficient for counterfactual decision-making.

Panel of Figure 5 — Figure 5: Rate verification for PW-OCP and DR-OCP. With $K=10$ , the left panel varies $T\in\{2{,}000,\ldots,50{,}000\}$ at fixed $\pi_{\min}=0.05$ , and the right panel varies $\pi_{\min}\in\{0.003,\ldots,0.05\}$ at fixed $T=20{,}000$ . Counterfactual coverage improves with the effective sample size $T\pi_{\min}$ , matching the dependence in Theorem 5.1 .

Figure 6: Step-size sensitivity. Counterfactual coverage gap is plotted against $\gamma$ for $K=5$ , $T=20{,}000$ , and $\pi_{\min}=0.05$ . The dashed vertical line marks $\gamma^{\star}=\sqrt{\pi_{\min}/T}\approx 1.58\times 10^{-3}$ . PW-OCP performs best near this scale, and both methods remain stable over a broad neighborhood of $\gamma^{\star}$ . Error bars are mean $\pm 95\%$ confidence intervals over $20$ seeds.

Figure 7: Real-data counterfactual coverage on Open Bandit and DJIA tasks. The three panels report the same counterfactual coverage-gap metric as Table 1 for OBD Women, OBD Men, and DJIA portfolio rebalancing. Absolute gaps are smaller than in the synthetic stress tests because several baselines already achieve near-nominal logged coverage, but PW-OCP and DR-OCP remain among the best non-oracle methods across all three benchmarks.

Explore similar work

Online Conformal Prediction Beyond Feedback
Aug 7, 2026Joar Skalse, Edoardo Pona, Osvaldo Simeone +1Uncertainty Quantification Conformal Prediction
King’s College London · Northeastern University London
Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction
Jul 2, 2026Yurui Zheng, Ying JinSet Prediction Conformal Prediction
School of Mathematical Sciences, Peking University · Department of Statistics and Data Science, University of Pennsylvania
Online Conformal Prediction with Corrupted Feedback
May 19, 2026Bowen Wang, Matteo Zecchin, Osvaldo SimeoneConformal Prediction Uncertainty Calibration
Department of Engineering, King’s College London, WC2R 2LS London, U.K. · Communication Systems Department, EURECOM, 06410 Sophia Antipolis, France · Institute for Intelligent Networked Systems, Northeastern University London, E1W 1LP London, U.K.

Aug 7, 2026J/K move · Enter open · S save

Online Conformal Prediction Beyond Feedback

Joar Skalse, Edoardo Pona, Osvaldo Simeone, Nicola Paoletti

King’s College London · Northeastern University London

Uncertainty Quantification Conformal Prediction

Open paper AskPDF arXiv

Online conformal prediction (OCP) provides theoretically principled uncertainty quantification for arbitrary black-box classifiers and non-i.i.d. data streams by constructing prediction sets that are guaranteed to contain the true label at a user-specified frequency. OCP usually updates prediction sets using feedback from previously deployed predictions. We instead study an OCP setting beyond feedback: on each round, the learner can either output a prediction set or query the correct label, but not both. Thus, no deployed prediction is ever evaluated directly. We reduce this problem to a partial monitoring game in which prediction actions return no observation and a separate query action reveals the label. To solve this game, we develop OCP with queries (OCPQ), a label-efficient forecasting algorithm designed for this beyond-feedback setting. Using an adequate reward construction, for any black-box classifier and any (non-i.i.d.) oblivious data stream of length

T

, OCPQ has

O(T^{2/3})

expected regret and expected coverage of at least

β-O(T^{-1/3})

for a user-defined

β

, while querying only an expected

T^{-1/3}

fraction of rounds. This provides coverage comparable to bandit-based OCP methods while requiring no feedback from deployed predictions. Experiments on real-world datasets demonstrate the effectiveness of our approach.

Abstract

Explore similar work

Online Conformal Prediction Beyond Feedback

Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction

Online Conformal Prediction with Corrupted Feedback