Multiple-Prediction-Powered Inference

Charlie Cowen-Breen; Alekh Agarwal; Stephen Bates; William W. Cohen; Jacob Eisenstein; Amir Globerson; Adam Fisch

math.STMar 28, 2026

Multiple-Prediction-Powered Inference

Authors: Charlie Cowen-Breen, Alekh Agarwal, Stephen Bates, William W. Cohen, Jacob Eisenstein, Amir Globerson, Adam Fisch

Organizations: Massachusetts Institute of Technology · Google Research · Google DeepMind

Abstract

Statistical estimation often involves tradeoffs between expensive, high-quality measurements and a variety of lower-quality proxies. We introduce Multiple-Prediction-Powered Inference (MultiPPI): a general framework for constructing statistically efficient estimates by optimally allocating resources across these diverse data sources. This work provides theoretical guarantees about the minimax optimality, finite-sample performance, and asymptotic normality of the MultiPPI estimator. Through experiments across three diverse large language model (LLM) evaluation scenarios, we show that MultiPPI consistently achieves lower estimation error than existing baselines. This advantage stems from its budget-adaptive allocation strategy, which strategically combines subsets of models by learning their complex cost and correlation structures.

Figures & tables

Panel of Figure 1 — Figure 1: Results by budget for the experiments on Chatbot Arena (a), ProcessBench (b), and Factuality (c). For each estimator (all baselines and MultiPPI), the left column plots the empirical coverage of the 95% CI, the middle column plots the width of the 95% CI, and the right column plots the empirical mean-squared error of the point estimate. The fully-labeled sample size $N$ is 250.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Proportion of budget allocated to different models in Experiment 1: ChatBot Arena. Gemini 2.5 Flash, the cheapest model, is most sampled in the low-budget regime, while the proportion of budget allocated to the joint (both models combined) increases monotonically with budget.

Figure 3: Proportion of budget allocated to different models in Experiment 2: ProcessBench. Tiny (125 word thinking budget) is most sampled in the low-budget regime, while the proportion of budget allocated to the joint (all models combined) increases monotonically with budget.

Figure 4: Linear parameters $\lambda_{I}$ learned across budget regimes in Experiment 2: ProcessBench. While only the tiny model (125 word thinking budget) has a nonzero linear parameter in the low-budget regime, a cascading behavior is learned in the large-budget regime: the cheaper models are prescribed the opposite sign from the more-expensive models in the joint term.

Figure 5: Linear parameters $\lambda_{I}$ learned across budget regimes in Experiment 1: ChatBot Arena. While only Gemini 2.5 Pro has a nonzero linear parameter in the low-budget regime, a cascading behavior is learned in the large-budget regime: the cheaper model (Gemini 2.5 Flash) is prescribed the opposite sign from the more-expensive model (Gemini 2.5 Pro) in the joint term.

Panel of Figure 6 — Figure 6: Results given only $N=10$ labeled examples. Results are shown by budget for the experiments on Chatbot Arena (top), ProcessBench (middle), and Factuality (bottom). For each estimator (all baselines and MultiPPI), the left column plots the empirical coverage of the 95% CI, the middle column plots the width of the 95% CI, and the right column plots the empirical mean-squared error of the point estimate.

Panel of Figure 7 — Figure 7: Results given $N=1000$ labeled examples. Results are shown by budget for the experiments on Chatbot Arena (top), ProcessBench (middle), and Factuality (bottom). For each estimator (all baselines and MultiPPI), the left column plots the empirical coverage of the 95% CI, the middle column plots the width of the 95% CI, and the right column plots the empirical mean-squared error of the point estimate.

Panel of Figure 8 — Figure 8: MultiPPI with varying number of fully labeled examples.

Panel of Figure 9 — Figure 9: Comparison of results with different techniques for covariance estimation, for $N=50$ . We find that Ledoit-Wolf shrinkage covariance estimation yields best performance in all regimes.

Panel of Figure 10 — Figure 10: Comparison of MultiPPI for $\mathcal{I}=2^{\{1,\dots,k\}}$ (default settings) with MultiPPI (Restricted), as defined in Section D.4 .

Figure 11: Performance at determination of process error vs. word budget. This is calculated via the procedure described in Appendix I . The majority of the improvement observed due to thinking occurs once 500 words of thought is reached, and plateaus around 1,000 words of thought.

Figure 12: Performance at determination of process error vs. word budget. This is calculated via the procedure described in Appendix I . The majority of the improvement observed due to thinking occurs once 500 words of thought is reached, and plateaus around 1,000 words of thought.

Figure 13: Performance at factuality evaluation with increasing number of agents and rounds of debate. Soft accuracy awards half a point to reporting an uncertain answer, while hard accuracy awards nothing.

Figure 14: Proportion of uncertain predictions by number of agents and rounds of debate. An increased number of agents leads to fewer uncertain predictions, and almost all predictions are certain by the end of the second round of debate.

Figure 15: Different schemes for evaluation with autoraters on the ProcessBench dataset. Gray: classical sampling—no autoraters. Orange: pure autoraters in increasing order of thinking budget (from top to bottom, word budget is 50, 500, 1000, 1500, 2000 and 3000) —note that the bias is increasingly pronounced with thinking budget. Green: various schemes for debiasing autoraters (from top to bottom, MultiPPI, vector PPI++, followed by scalar PPI++ with 3000, 2000, 1500, and 1000 word thinking budget, respectively).

Model collection	Cost
Gemini 2.5 Pro	$1.25
Gemini 2.5 Flash	$0.30
Both	$1.55

Appendix

Table 1: Cost structure for experiment 1.

Figure 16: Prompt used to generate autoratings for Experiment 2.

Figure 17: Depiction of biography-fact pairs $(b,f)$ as in Experiment 3. Judgements about factual consistency of $(b,f)$ are made by a language model.

Prediction-Powered Inference

Prediction-Powered Inference uses model predictions and a smaller set of trusted labels to make efficient estimates with statistically valid uncertainty.

See all 30 papers

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data
Jun 29, 2026Haobo YangLanguage Model Calibration Statistical Learning Theory
Department of Computer Science and Engineering, SUSTech University
Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
May 11, 2026Elad Tolochinsky, Yaniv Tenzer, Yaniv RomanoModel Selection Multi-Armed Bandits
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology
Instance-Optimal Estimation with Multiple LLM Judges on a Budget
May 22, 2026Junghyun Lee, Sanghwa Kim, Yassir Jedra +2Parameter Estimation LLM Evaluation
KAIST AI · ICL EEE · KTH EECS

Jun 29, 2026J/K move · Enter open · S save

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data

Haobo Yang

Department of Computer Science and Engineering, SUSTech University

Language Model Calibration Statistical Learning Theory

Open paper AskPDF arXiv

Quantitative research across the social and behavioral sciences depends on human subject experiments that are expensive, slow, and subject to sampling bias. Here we show that pretrained large language models induce risk-equivalent estimators of conditional expectations under squared loss, establishing restricted functional risk equivalence: under squared loss, the LLM induces an estimator whose risk matches the Bayes optimal risk for squared-loss prediction of conditional expectations for any inference that depends on the data only through the conditional mean. We formalize the LLM as a misspecified functional estimator

T(\hat{P}_n)

trained on i.i.d.\ data, decompose the estimation error into representation bias

ε_{\mathrm{rep}}

and optimization error, and prove that under mild regularity conditions the LLM's expected error converges to the irreducible population variance plus the squared representation bias, with the representation bias bounded by the Pinsker inequality. The identifiability error

δ

propagates into the effective bias, inflating the asymptotic risk floor. We establish restricted functional risk equivalence via a bidirectional Le Cam deficiency analysis: the forward deficiency vanishes asymptotically while the reverse deficiency is exactly zero. We provide finite-sample concentration bounds and a calibration protocol with explicit decision rules. The result is a precise, provable statement: a well-calibrated LLM achieves the Bayes-optimal risk for conditional-mean-dependent inference, bounded by explicit scope conditions. In practical applications, this means that under satisfied conditions and well-calibrated models, large language models can be used in many prediction and decision-making tasks that originally relied on human experiments, approximating near-optimal statistical inference at lower cost.

Abstract

Prediction-Powered Inference

Explore similar work

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Instance-Optimal Estimation with Multiple LLM Judges on a Budget