Pro-Active Inference

Skyler Wu; Yash Nair; Antoine Maechler; Emmanuel J. Candès

stat.MEOct 26, 2026

Pro-Active Inference

Authors: Skyler Wu, Yash Nair, Antoine Maechler, Emmanuel J. Candès

Organizations: Department of Statistics, Stanford University · Department of Management Science and Engineering, Stanford University

Abstract

Estimating a population mean is fundamental across scientific applications, but observing every outcome can be prohibitively expensive. Often, covariates and machine learning (ML) predictions are available for the full population, even when only a small subset of outcomes can be observed. How can we leverage this information to proactively select observations to label, update ML predictions as labels arrive, and construct statistically valid confidence intervals? We develop a general, without-replacement formulation of Pro-Active Inference (PAI), extending our earlier with-replacement approach and building on Active Statistical Inference. Rather than making fixed-order label-skip decisions, PAI proactively selects which observation to label next from the remaining covariate pool while accommodating online ML updates. We establish PAI's finite-sample unbiasedness and asymptotic validity, develop an improved finite-sample variance estimator, and extend PAI to superpopulation inference without Bonferroni corrections or $α$ -splitting. We characterize PAI's potential efficiency gains over Batch Active Inference in a quasi-oracle setting, highlighting the role of prediction-residual imbalance. Across eight diverse datasets, PAI achieves greater sampling efficiency than Sequential Active Inference in the vast majority of tested settings with near-nominal statistical coverage, using the same ML predictor and uncertainty score. However, comparisons with simpler baselines such as Prediction-Powered Inference and uniform sampling, as well as the closely related Prediction-Powered Active Testing, reveal when additional adaptivity helps or hurts statistical efficiency. Through interpretable, theory-motivated diagnostics and frozen-ML ablations, we investigate when and why PAI gains or loses efficiency, highlighting the initial residual-estimation problem and subsequent online learning dynamics.

Figures & tables

Figure 1: Comparing sampling efficiency of Pro-Active Inference (PAI), Sequential Active Inference (SAI), and Prediction-Powered Inference (PPI) with respect to uniform sampling across seven datasets. Each subplot shows the effective sample size (ESS) multiplier with respect to uniform sampling as a function of labeling budget for PAI, SAI, and PPI. For example, on ACS at a

10\%

budget, an ESS multiplier of

1.6

means PAI matches the confidence interval width of uniform sampling at a

16\%

budget. Standard errors are shown, but negligible on the plot scale. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating an improvement over uniform sampling.

Figure 2: ESS multipliers of PAI over SAI as a function of initial residual imbalance

b_{0}^{2}

. Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. Unlike Figure 1 , ESS multipliers here are measured with respect to SAI rather than uniform sampling. For example, an ESS multiplier of

2.5

means that PAI matches the confidence interval width of SAI while using only

40\%

as many labels. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that PAI outperforms SAI. Spearman rank correlations were computed across the seven datasets at each fixed budget, and we report the median, interquartile range (IQR), and minimum of the resulting ten correlations.

Figure 3: ESS multipliers of PAI over PPAT across seven datasets. Each subplot shows PAI’s ESS multiplier with respect to PPAT as a function of labeling budget. For example, on AlphaFold at a

2.5\%

budget, an ESS multiplier of

1.3

means PAI matches the confidence interval width of PPAT at a

3.25\%

budget. Standard errors are shown but are generally negligible at the plot scale. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that PAI outperforms PPAT.

Figure 4: ESS multipliers of PAI over PPI and PPAT as a function of the expected

1

-step relative change in Brier score (ECB). Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. The left and right subplots show ESS multipliers with respect to PPI and PPAT, respectively. For example, on ACS in the left subplot, an ESS multiplier of

1.25

means that PAI matches the confidence interval width of PPI while using

20\%

fewer labels. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that PAI outperforms the corresponding competitor. Spearman rank correlations were computed across the six datasets at each fixed budget, and we report the minimum and maximum of the resulting ten correlations in each subplot.

Figure 5: ESS multipliers of PAI over uniform sampling as a function of initial variance ratio (IVR). Each color represents one dataset, with ten points corresponding to the ten labeling budgets tested. An

\text{IVR}(n)

greater than

1

indicates that the initial variance approximation favors PAI over uniform sampling, suggesting a potential efficiency advantage for PAI. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that PAI outperforms uniform sampling. Spearman rank correlations were computed across the seven datasets at each fixed budget, and we report the minimum and maximum of the resulting ten correlations.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Comparing empirical coverages of Pro-Active Inference (PAI), Sequential Active Inference (SAI), Prediction-Powered Inference (PPI), Uniform Sampling (Uniform), and Prediction-Powered Active Testing (PPAT) across seven datasets. Each subplot shows the empirical coverage of nominal

95\%

normal-approximation confidence intervals as a function of labeling budget. For all datasets except the two LLM evaluation suites (MMLU-Pro and BBH+GPQA+IFEval+MATH+MusR), empirical coverages are computed over

1000

random seeds. For the two LLM evaluation suites, empirical coverages are averaged over

2.2

K test models per evaluation suite and

100

seeds for computational efficiency. Standard errors are shown, but are negligible at the plotted scale.

Figure 7: ESS multipliers of without-replacement PAI over with-replacement PAI across the two LLM evaluation suites. Each subplot shows the ESS multiplier of without-replacement PAI with respect to with-replacement PAI as a function of labeling budget. For example, on MMLU-Pro at a

22.5\%

budget, an ESS multiplier of

1.3

means that without-replacement PAI matches the confidence interval width of with-replacement PAI at a

29.25\%

budget. Standard errors are shown, but are negligible at the plotted scale. Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that without-replacement PAI outperforms with-replacement PAI.

Figure 8: ESS multipliers of PAI and its frozen-ML ablation over PPI, PPAT, and uniform sampling across six datasets. Each row corresponds to one competitor (PPI, PPAT, or Uniform) and each column corresponds to one dataset. Each subplot shows the ESS multipliers of standard PAI (blue triangles) and frozen-ML PAI (red circles) with respect to the corresponding competitor method as a function of labeling budget. In the frozen-ML ablation, the initial ML model is held fixed throughout sampling, affecting both imputation and acquisition. The competing methods remain unchanged from their specifications in Section 5 . Higher ESS multipliers indicate better sampling efficiency, with values above

1

indicating that the corresponding PAI variant outperforms the competitor.

Adaptive Sampling

Adaptive sampling allocates limited data, measurements, or computation where they are expected to be most useful.

See all 205 papers

Learning U-Statistics with Active Inference
May 12, 2026Xiaoning Wang, Yuyang Huo, Liuhua Peng +1Active Inference U-Statistics
School of Statistics and Data Sciences, LPMC, KLM-DASR and LEBPS, Nankai University, Tianjin, China · School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia
Active Multiple-Prediction-Powered Inference
May 8, 2026Nicholas Brawand, Nima Leclerc, Anhthy Ngo +4Adaptive Inference Cost-Aware Inference
The MITRE Corporation · Georgia Institute of Technology
Prediction-Powered Active Testing
Jul 9, 2026Kianoosh Ashouritaklimi, Valentin Kilian, Daolang Huang +2Confidence Region Estimation Variance Reduction
Department of Statistics, University of Oxford · Department of Computer Science, Aalto University

May 12, 2026J/K move · Enter open · S save

Learning U-Statistics with Active Inference

Xiaoning Wang, Yuyang Huo, Liuhua Peng, Changliang Zou

School of Statistics and Data Sciences, LPMC, KLM-DASR and LEBPS, Nankai University, Tianjin, China · School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia

Active Inference U-Statistics Active Learning Adaptive Sampling Statistical Learning Theory Empirical Risk Minimization

Open paper AskPDF arXiv

U

-statistics play a central role in statistical inference. In many modern applications, however, acquiring the labels required for

U

-statistics is costly. Motivated by recent advances in active inference, we develop an active inference framework for

U

-statistics that selectively queries informative labels to improve estimation efficiency under a fixed labeling budget, while preserving valid statistical inference. Our approach is built on the augmented inverse probability weighting

U

-statistic, which is designed to incorporate the sampling rule and machine learning predictions. We characterize the optimal sampling rule that minimizes its variance and design practical sampling strategies. We further extend the framework to

U

-statistic-based empirical risk minimization. Experiments on real datasets demonstrate substantial gains in estimation efficiency over baseline methods, while maintaining target coverage.

Abstract

Adaptive Sampling

Explore similar work

Learning U-Statistics with Active Inference

Active Multiple-Prediction-Powered Inference

Prediction-Powered Active Testing