Modern studies increasingly leverage outcomes predicted by machine learning and artificial intelligence (AI/ML) models, and recent work, such as prediction-powered inference (PPI), has developed valid downstream statistical inference procedures. However, classical power and sample size formulas do not readily account for these predictions. In this work, we tackle a simple yet practical question: given a new AI/ML model with high predictive power, how many labeled samples are needed to achieve a desired level of statistical power? We derive closed-form power formulas by characterizing the asymptotic variance of the PPI estimator and applying Wald test inversion to obtain the required labeled sample size. Our results cover widely used settings including two-sample comparisons and risk measures in 2x2 tables. We find that a useful rule of thumb is that the reduction in required labeled samples relative to classical designs scales roughly with the R2 between the predictions and the ground truth. Our analytical formulas are validated using Monte Carlo simulations, and we illustrate the framework in three contemporary biomedical applications spanning single-cell transcriptomics, clinical blood pressure measurement, and dermoscopy imaging. We provide our software as an R package and online calculators at https://github.com/yiqunchen/pppower.
Figures & tables
Figure 1: Prediction-powered planning. Panel (a) shows the notation, estimator, and variance inversion. Panel (b) shows the NHANES sample-size plan for Δ=4 mmHg: gray is the classical design, and the orange curves are the age-only linear model, the richer clinical linear model, and the random forest surrogate.
Figure 2: One-sample mean validation: empirical power (points, with 95% Monte Carlo error bars for the binary setting) versus theoretical power (lines) for PPI++ with oracle λ⋆ .
Figure 3: Odds-ratio and relative-risk validation in 2×2 tables—analytical (lines) versus empirical (points) PPI++ power for odds ratio (top) and relative risk (bottom). Dashed lines show classical power.
Figure 4: Regression-contrast validation: analytical (lines) versus empirical (points) PPI++ power. In panel (b), the empirical logistic-regression points use a two-fold cross-fitted plug-in λ^ estimated within each replicate, while the analytical curve uses the large-reference-sample approximation. Dashed lines show the corresponding classical power curves.
Figure 5: Real-data planning and held-out validation across three biomedical case studies. Each row corresponds to one application (Baron scRNA-seq, NHANES systolic blood pressure, and ISIC melanoma), and the three columns give a study overview, the planned labeled sample size, and the held-out achieved power.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Design
Method
Required n
Achieved power
Type I error
Reduction
One-sample
Classical
197
0.797
0.048
0.0%
One-sample
Vanilla PPI
123
0.801
0.052
37.6%
One-sample
PPI++ (Oracle)
102
0.811
0.048
48.2%
One-sample
PPI++ (Plugin)
102
0.804
0.056
48.2%
Two-sample
Classical
175
0.796
0.046
0.0%
Two-sample
Vanilla PPI
109
0.791
0.048
37.7%
Appendix
Table S1 : Head-to-head comparison across representative continuous planning scenarios (one-sample: Δ=0.2 ; two-sample: Δ=0.3 ; paired: Δ=0.3 ; all with N=5000 , ρ=0.7 , target power 0.80 , α=0.05 ; MC R=5000 ). For plugin PPI++ , planning uses the oracle closed-form n and evaluates performance with plugin λ^ .
Application
Surrogate
ρYf2
Δ
n⋆
Achieved power
Reduction
Baron
Classical
—
0.10
785
0.840
—
Baron
Donor GLM
0.141
0.10
740
0.826
6%
Baron
INS GLM
0.967
0.10
325
0.826
59%
Baron
Top-20 RF
0.983
0.10
306
0.840
61%
NHANES
Classical
—
4 mmHg
187
0.830
—
NHANES
Age-only LM
0.218
4 mmHg
149
0.836
20%
Appendix
Table S2 : Representative planning outputs and held-out achieved power from the 15% pilot / 60% held-out workflow. The Baron row uses the beta-minus-alpha composition contrast at N=500 ; NHANES uses mean systolic blood pressure at N=2,000 ; and ISIC uses melanoma prevalence at N=10,000 .
Figure S1 : Two-sample t -test with continuous Gaussian outcomes. Analytical (lines) versus empirical (points) power.
Figure S2 : Two-sample proportion test with binary outcomes. Analytical (lines) versus empirical (points) power.
Figure S3 : Paired t -test with continuous differences. Analytical (lines) versus empirical (points) power.
Figure S4 : Paired proportion test with binary differences. Analytical (lines) versus empirical (points) power.
Figure S5 : Sample-size inversion check across one-sample, two-sample, and paired designs. Points show achieved power minus target power at the planned integer n⋆ ; exact inversion would lie on the horizontal zero line.
Figure S6 : Rule-of-thumb error for the labeled-sample reduction ratio. Curves plot nPPI/ncl−(1−ρ2) against ρ2 for several unlabeled-sample sizes N .
Figure S7 : Type I error calibration for the continuous null settings. Rejection rates remain close to the nominal level 0.05, with dotted lines showing the corresponding Monte Carlo fluctuation band for R=2,000 replicates.
Figure S8 : Convergence of the plugin λ^ toward the oracle λ⋆ as the labeled sample size grows. The plotted quantity is the root-mean-squared error relative to the oracle tuning value.
Figure S9 : Power with plugin versus oracle tuning. The analytical curve uses the oracle variance formula, while the empirical points compare oracle and plugin versions of the PPI++ test.
Figure S10 : Power as a function of effect size for the Gaussian one-sample mean problem. Higher prediction quality shifts the PPI++ curve leftward relative to the classical design.
Figure S11 : Small- n regime for the Gaussian one-sample mean problem. Agreement remains usable down to n=15 , with the largest discrepancies concentrated in the smallest labeled samples.
Figure S12 : Sensitivity to the unlabeled-to-labeled ratio N/n . Power gains increase quickly at first and then saturate once the unlabeled pool is moderately large.
Figure S13 : Unequal-group two-sample designs. Power decreases with stronger allocation imbalance, while the analytical formula continues to track the empirical results closely.
Figure S14 : Sensitivity to misspecified prediction quality. The dashed line marks the target power, while each panel varies the true prediction quality around the value used at the planning stage.
Figure S15 : Distributional robustness for the one-sample mean problem. Top: log-normal outcomes, where the CLT-based formula overshoots at small n under severe skew (max discrepancy 0.144 at n=50 , ρ=0.5 ). Bottom: t5 outcomes, where agreement remains tight despite heavy tails (max discrepancy 0.018).
Sample size determination for machine learning (ML) prediction models is challenging because conventional power analysis typically requires the predictor-outcome relationship and effect structure to be specified a priori. Nonlinear ML models learn complex prediction surfaces that do not admit straightforward analytical power calculations. We propose a framework that approximates nonlinear ML models with localized linear representations and estimates sample size requirements by evaluating statistical power across these local regions.
Dat Phan-Trong, Sunil Gupta, Svetha Venkatesh
Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia
Many applications require statistically valid inference across many related tasks, while using only a handful of high-quality labels per hypothesis. In AI evaluation, these tasks may correspond to model behaviors across prompts, subgroups, or hypotheses; in social science surveys, they may correspond to related questions, populations, or measurement conditions. Prediction-powered inference (PPI) uses abundant but inexpensive proxy measurements to improve inference from limited, ground-truth labels, but commonly used methods treat tasks independently and therefore fail to exploit shared structure across related tasks. This limitation is especially important in settings where only a small number of labels are available per task. To address this issue, we introduce a multi-task prediction-powered inference framework that uses labeled data from related tasks to improve power while preserving task-specific inference. Our methods exploit the shared structure in the proxy-ground-truth relationship through cross-task recalibration, while retaining within-task rectification and power tuning to construct accurate point estimates and confidence intervals. We prove that efficiency gains beyond power-tuned PPI are only possible when the proxy-ground-truth relationship contains nonlinear structure; affine cross-task recalibrations are asymptotically equivalent to using the original proxy. We complement our theoretical findings with experiments on synthetic and semi-synthetic datasets, as well as a case study auditing language models on election-related information during the 2024 U.S. presidential election. Using a large human-annotation study, we show that cross-task recalibration can substantially reduce confidence interval widths when labels are scarce.
Nicolas Emmenegger, Ellery Stahler, Chara Podimata
We establish a formal connection between the decades-old surrogate outcome model in biostatistics and economics and the emerging field of prediction-powered inference (PPI). The connection treats predictions from pre-trained models, prevalent in the age of AI, as cost-effective surrogates for expensive outcomes. Building on the surrogate outcomes literature, we develop recalibrated prediction-powered inference, a more efficient approach to statistical inference than existing PPI proposals. Our method departs from the existing proposals by using flexible machine learning techniques to learn the optimal ``imputed loss'' through a step we call recalibration. Importantly, the method always improves upon the estimator that relies solely on the data with available true outcomes, even when the optimal imputed loss is estimated imperfectly, and it achieves the smallest asymptotic variance among PPI estimators if the estimate is consistent. Computationally, our optimization objective is convex whenever the loss function that defines the target parameter is convex. We further analyze the benefits of recalibration, both theoretically and numerically, in several common scenarios where machine learning predictions systematically deviate from the outcome of interest. We demonstrate significant gains in effective sample size over existing PPI proposals via three applications leveraging state-of-the-art machine learning/AI models.