Estimating a population mean is fundamental across scientific applications, but observing every outcome can be prohibitively expensive. Often, covariates and machine learning (ML) predictions are available for the full population, even when only a small subset of outcomes can be observed. How can we leverage this information to proactively select observations to label, update ML predictions as labels arrive, and construct statistically valid confidence intervals? We develop a general, without-replacement formulation of Pro-Active Inference (PAI), extending our earlier with-replacement approach and building on Active Statistical Inference. Rather than making fixed-order label-skip decisions, PAI proactively selects which observation to label next from the remaining covariate pool while accommodating online ML updates. We establish PAI's finite-sample unbiasedness and asymptotic validity, develop an improved finite-sample variance estimator, and extend PAI to superpopulation inference without Bonferroni corrections or α-splitting. We characterize PAI's potential efficiency gains over Batch Active Inference in a quasi-oracle setting, highlighting the role of prediction-residual imbalance. Across eight diverse datasets, PAI achieves greater sampling efficiency than Sequential Active Inference in the vast majority of tested settings with near-nominal statistical coverage, using the same ML predictor and uncertainty score. However, comparisons with simpler baselines such as Prediction-Powered Inference and uniform sampling, as well as the closely related Prediction-Powered Active Testing, reveal when additional adaptivity helps or hurts statistical efficiency. Through interpretable, theory-motivated diagnostics and frozen-ML ablations, we investigate when and why PAI gains or loses efficiency, highlighting the initial residual-estimation problem and subsequent online learning dynamics.
Figures & tables
Figure 1: Comparing sampling efficiency of Pro-Active Inference (PAI), Sequential Active Inference (SAI), and Prediction-Powered Inference (PPI) with respect to uniform sampling across seven datasets. Each subplot shows the effective sample size (ESS) multiplier with respect to uniform sampling as a function of labeling budget for PAI, SAI, and PPI. For example, on ACS at a 10% budget, an ESS multiplier of 1.6 means PAI matches the confidence interval width of uniform sampling at a 16% budget. Standard errors are shown, but negligible on the plot scale. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating an improvement over uniform sampling.
Figure 2: ESS multipliers of PAI over SAI as a function of initial residual imbalance b02 . Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. Unlike Figure 1 , ESS multipliers here are measured with respect to SAI rather than uniform sampling. For example, an ESS multiplier of 2.5 means that PAI matches the confidence interval width of SAI while using only 40% as many labels. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that PAI outperforms SAI. Spearman rank correlations were computed across the seven datasets at each fixed budget, and we report the median, interquartile range (IQR), and minimum of the resulting ten correlations.
Figure 3: ESS multipliers of PAI over PPAT across seven datasets. Each subplot shows PAI’s ESS multiplier with respect to PPAT as a function of labeling budget. For example, on AlphaFold at a 2.5% budget, an ESS multiplier of 1.3 means PAI matches the confidence interval width of PPAT at a 3.25% budget. Standard errors are shown but are generally negligible at the plot scale. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that PAI outperforms PPAT.
Figure 4: ESS multipliers of PAI over PPI and PPAT as a function of the expected 1 -step relative change in Brier score (ECB). Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. The left and right subplots show ESS multipliers with respect to PPI and PPAT, respectively. For example, on ACS in the left subplot, an ESS multiplier of 1.25 means that PAI matches the confidence interval width of PPI while using 20% fewer labels. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that PAI outperforms the corresponding competitor. Spearman rank correlations were computed across the six datasets at each fixed budget, and we report the minimum and maximum of the resulting ten correlations in each subplot.
Figure 5: ESS multipliers of PAI over uniform sampling as a function of initial variance ratio (IVR). Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. An IVR(n) greater than 1 indicates that the initial variance approximation favors PAI over uniform sampling, suggesting a potential efficiency advantage for PAI. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that PAI outperforms uniform sampling. Spearman rank correlations were computed across the seven datasets at each fixed budget, and we report the minimum and maximum of the resulting ten correlations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Comparing empirical coverages of Pro-Active Inference (PAI), Sequential Active Inference (SAI), Prediction-Powered Inference (PPI), Uniform Sampling (Uniform), and Prediction-Powered Active Testing (PPAT) across seven datasets. Each subplot shows the empirical coverage of nominal 95% normal-approximation confidence intervals as a function of labeling budget. For all datasets except the two LLM evaluation suites (MMLU-Pro and BBH+GPQA+IFEval+MATH+MusR), empirical coverages are computed over 1000 random seeds. For the two LLM evaluation suites, empirical coverages are averaged over 2.2 K test models per evaluation suite and 100 seeds for computational efficiency. Standard errors are shown, but are negligible at the plotted scale.
Figure 7: ESS multipliers of without-replacement PAI over with-replacement PAI across the two LLM evaluation suites. Each subplot shows the ESS multiplier of without-replacement PAI with respect to with-replacement PAI as a function of labeling budget. For example, on MMLU-Pro at a 22.5% budget, an ESS multiplier of 1.3 means that without-replacement PAI matches the confidence interval width of with-replacement PAI at a 29.25% budget. Standard errors are shown, but are negligible at the plotted scale. Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that without-replacement PAI outperforms with-replacement PAI.
Figure 8: ESS multipliers of PAI and its frozen-ML ablation over PPI, PPAT, and uniform sampling across six datasets. Each row corresponds to one competitor (PPI, PPAT, or Uniform) and each column corresponds to one dataset. Each subplot shows the ESS multipliers of standard PAI (blue triangles) and frozen-ML PAI (red circles) with respect to the corresponding competitor method as a function of labeling budget. In the frozen-ML ablation, the initial ML model is held fixed throughout sampling, affecting both imputation and acquisition. The competing methods remain unchanged from their specifications in Section 5 . Higher ESS multipliers indicate better sampling efficiency, with values above 1 indicating that the corresponding PAI variant outperforms the competitor.
U-statistics play a central role in statistical inference. In many modern applications, however, acquiring the labels required for U-statistics is costly. Motivated by recent advances in active inference, we develop an active inference framework for U-statistics that selectively queries informative labels to improve estimation efficiency under a fixed labeling budget, while preserving valid statistical inference. Our approach is built on the augmented inverse probability weighting U-statistic, which is designed to incorporate the sampling rule and machine learning predictions. We characterize the optimal sampling rule that minimizes its variance and design practical sampling strategies. We further extend the framework to U-statistic-based empirical risk minimization. Experiments on real datasets demonstrate substantial gains in estimation efficiency over baseline methods, while maintaining target coverage.
Xiaoning Wang, Yuyang Huo, Liuhua Peng +1
School of Statistics and Data Sciences, LPMC, KLM-DASR and LEBPS, Nankai University, Tianjin, China · School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia
Post-deployment monitoring of healthcare AI requires statistically valid, label-efficient methods, but gold-standard labels from clinician chart review are expensive. Prediction-powered inference (PPI) and active statistical inference (ASI) reduce label cost by combining a small labeled sample with abundant model predictions, but both are restricted to a single predictor, a poor fit for modern clinical pipelines that have multiple predictors of differing cost and accuracy available at inference time. We propose Active Multiple-Prediction-Powered Inference (AM-PPI), which routes each instance to a cost-appropriate predictor subset, samples gold-standard labels in proportion to the chosen subset's residual uncertainty, and reweights predictions to minimize estimator variance, all under a single deployment-time budget. AM-PPI generalizes ASI to leverage multiple predictors and extends Multiple-PPI from global per-predictor allocation to per-instance adaptive routing. We derive closed-form Karush-Kuhn-Tucker (KKT) conditions for all three decisions and prove, via biconvexity and strong duality, that the resulting fixed point is a global optimum despite the joint problem being non-jointly-convex. We establish asymptotic normality with valid coverage, minimum-variance unbiasedness within the linear-prediction augmented inverse propensity weighted (AIPW) class, and a closed-form criterion identifying when multiple predictors help. On synthetic data and three healthcare monitoring tasks, AM-PPI produces 10 to 40 percent narrower confidence intervals (CIs) than single-predictor ASI in the budget regime where routing matters, and matches the better baseline elsewhere.
Nicholas Brawand, Nima Leclerc, Anhthy Ngo +4
The MITRE Corporation · Georgia Institute of Technology
Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose Prediction--Powered Active Testing (PPAT), a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.