Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation

Dae Woong; Ham; Xuejun Zhao; Stefanus Jasin; Fenghua Yang

cs.AISep 24, 2026

Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation

Authors: Dae Woong, Ham, Xuejun Zhao, Stefanus Jasin, Fenghua Yang

Organizations: David · University of Michigan, Ann Arbor, MI 48109, USA · University of North Carolina at Charlotte, Charlotte, NC 28223, USA

Abstract

Large language models are increasingly used as inexpensive judges to evaluate outputs, label data, and assess whether a system meets a desired quality standard. Yet using AI judgments for formal statistical inference is fundamentally different from simply treating them as ground-truth labels: AI evaluations can be biased or noisy, and rigorous hypothesis testing requires explicit control of type-I and type-II errors. We study how to use AI judgments, together with selective human verification, to conduct a valid hypothesis test at minimum cost. We consider a population of items with hidden binary labels. After choosing a fixed pool of items, the decision maker can selectively query AI, send an item directly to a human, escalate an AI-scored item to a human after observing the AI report, or stop once sufficient evidence has accumulated. We derive an information-theoretic lower bound that captures the minimum cost of achieving prescribed testing errors and characterizes the value of AI information and human verification through a report-dependent information frontier. Motivated by this characterization, we develop SCALE, a sequential cost-aware policy that combines selective AI scoring with adaptive human escalation. SCALE is valid at finite sample sizes and matches the lower bound to first order as the target error probabilities vanish. We further extend the framework to an unknown AI-output model using paired AI-human pilot data. Numerically, SCALE approaches Human-only or AI-only testing when one source clearly dominates, while achieving its largest savings when inexpensive AI judgments and selective human verification are both valuable.

Figures & tables

Tuning parameter	Meaning	Choice
$f_{\mathrm{fb},k}$	Fallback-budget fraction	Any constant in $(0,1)$ bounded away from $0$ and $1$
$z_{k}$	Hypothesis boundary	Any $z_{k}\geq 0$ with $(z_{k}+1)/\Delta_{k}\to 0$ (see ( 42 )) where $\Delta_{k}$ satisfies $\Delta_{k}\to\infty$ , $\Delta_{k}=o(L_{k})$ , and $\Delta_{k}^{2}/L_{k}\to\infty$ (see ( 41 ))
$N_{\mathrm{main},k}$	Fixed pool size	$N_{\mathrm{main},k}=\overline{N}_{\mathrm{main},k}+1$ , where $\overline{N}_{\mathrm{main},k}\in\operatorname*{arg\,min}\bigl\{F^{\Delta}_{k}(N):N\in\mathbb{Z}_{+},\,N\geq N_{\mathrm{fixed},\mathrm{H}}(\alpha_{2,k},\beta_{2,k})\bigr\}$ and $F^{\Delta}_{k}(N)=Nc_{\mathrm{data}}+\max\{\Gamma_{1}(T_{1,k},N),\Gamma_{0}(T_{0,k},N)\}$ (see ( 33 )–( 35 ))
$\eta^{\mathrm{H}}_{h,k}$ , $h\in\{0,1\}$	The probability to directly query human	$\eta^{\mathrm{H}}_{h,k}=\lceil n^{}_{\mathrm{H},h,k}\rceil/(\lceil n^{}_{\mathrm{H},h,k}\rceil+\lceil n^{*}_{\mathrm{AI},h,k}\rceil)$ , and $0$ if the denominator vanishes
$\eta^{\mathrm{esc}}_{h,k}(\cdot)$ , $h\in\{0,1\}$	The probability to escalate to human given that the item has already been AI-queried before	Optimal solution of ( 16 ) with $s=s_{h,k}$ (see ( 39 ))
$\eta^{\mathrm{H}}_{,k}$ , $\eta^{\mathrm{esc}}_{,k}(\cdot)$	Dead-zone sensing rule	$\eta^{\mathrm{H}}_{,k}=(\eta^{\mathrm{H}}_{0,k}+\eta^{\mathrm{H}}_{1,k})/2$ and $\eta^{\mathrm{esc}}_{,k}(r)=(\eta^{\mathrm{esc}}_{0,k}(r)+\eta^{\mathrm{esc}}_{1,k}(r))/2$ , $r\in\mathcal{R}$ (see ( 40 ))

Table 1: Tuning parameters of the policy

\pi_{k}

in Algorithm 1 .

Parameter	Meaning	Value
$f_{\mathrm{fb},k}$ , $z_{k}$	Fallback-budget fraction and hypothesis boundary	Same as in Table 1
$\widehat{N}_{\mathrm{main},m,k}$	Fixed pool size	$\widehat{N}_{\mathrm{main},m,k}=\widehat{\overline{N}}_{\mathrm{main},m,k}+1$ , where $\widehat{\overline{N}}_{\mathrm{main},m,k}\in\operatorname*{arg\,min}\bigl\{\widehat{F}^{\mathrm{pilot}}_{m,k}(N):N\in\mathbb{Z}_{+},\,N\geq N_{\mathrm{fixed},\mathrm{H}}(\alpha_{2,k},\beta_{2,k})\bigr\}$ and $\widehat{F}^{\mathrm{pilot}}_{m,k}(N)=Nc_{\mathrm{data}}+\max_{h}\hat{\Gamma}_{h}(\widehat{T}^{\mathrm{pilot}}_{h,m,k}(N),N)$ (see ( 53 )–( 55 )); if problem ( 55 ) is not feasible, let $\widehat{N}_{\mathrm{main},m,k}=N_{\mathrm{fixed},\mathrm{H}}(\alpha_{2,k},\beta_{2,k})$
$a_{k}+\omega_{m,k},b_{k}+\omega_{m,k}$	Guarded stopping boundaries	See $a_{k},b_{k}$ in ( 25 ); $\omega_{m,k}=\widehat{N}_{\mathrm{main},m,k}\varepsilon_{m}^{\mathrm{LR}}$ when problem ( 55 ) is feasible, and $\omega_{m,k}=0$ otherwise
$\hat{\eta}^{\mathrm{H}}_{h,m,k},h\in\{0,1\}$	The probability to directly query human	See ( 56 ) if problem ( 55 ) is feasible; otherwise let $\hat{\eta}^{\mathrm{H}}_{h,m,k}=1$
$\hat{\eta}^{\mathrm{esc}}_{h,m,k}(\cdot),h\in\{0,1\}$	The probability to escalate to human given that the item has already been AI-queried before	Optimal solution of ( 50 ) with $\hat{s}_{h,m,k}=\hat{n}^{}_{\mathrm{esc},h,m,k}/\hat{n}^{}_{\mathrm{AI},h,m,k}$ if problem ( 55 ) is feasible; otherwise let $\hat{\eta}^{\mathrm{esc}}_{h,m,k}(\cdot)=0$
$\hat{\eta}^{\mathrm{H}}_{,m,k}$ , $\hat{\eta}^{\mathrm{esc}}_{,m,k}(\cdot)$	Dead-zone rule	Averages of the $h=0$ and $h=1$ rules, as in ( 40 )

Table 2: Parameters of the guarded plug-in policy

\hat{\pi}_{m,k}

in Algorithm 2 .

Explore similar work

Sep 21, 2026cs.LG

Augmented Hypothesis Testing with Persona-Based LLM Simulations

A/B testing requires large sample sizes, long timelines, and significant costs. When auxiliary predictions of experimental outcomes are available from machine learning models, uncertain prediction quality precludes replacing human experiments entirely, yet these predictions may still contain useful signal. We propose a principled framework for learning-augmented hypothesis testing that leverages predictions of unknown quality to reduce sample sizes while maintaining statistical validity. Predictions naturally vary in granularity, from coarse aggregate signals to fine-grained individual-level estimates, and our framework addresses both ends of this spectrum: (1) for population-level directional predictions, where only a binary signal on the treatment effect sign is available, we use an asymmetric test and prove consistency and robustness bounds within the learning-augmented algorithms paradigm; (2) for individual-level predictions, we introduce Generalized PPI++ (GPPI), extending Prediction-Powered Inference to handle nonlinear prediction errors through higher-dimensional transformations. Both methods benefit from accurate predictions while remaining robust to inaccurate or adversarial ones. We validate our framework using persona-based LLM simulations, where AI agents equipped with user personas predict individual behavior, as a natural prediction source spanning both granularity levels. Experiments on four real-world datasets demonstrate that our methods, combined with persona-based predictions, substantially reduce experimental costs while preserving rigorous statistical validity.

Ziyad Benomar, Aymen Al Marjani, Paul Missault +1

Amazon Luxembourg, Luxembourg · Amazon Barcelona, Spain

Two-Sample Testing User Simulation Prediction Error Human-In-The-Loop Clinical Prediction

Ask PDF

Aug 7, 2026cs.AI

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Jiacheng Miao, Jin Mu, Guanhua Chen +1

Department of Biomedical Data Science, Stanford University · Department of Genetics, Stanford University · Department of Biostatistics and Medical Informatics, University of Wisconsin–Madison +3

Two-Sample Testing Large Language Model Agents Fisher Information

Ask PDF

May 27, 2026cs.LG

Semi-Supervised Hypothesis Testing by Betting on Predictions

We introduce a testing-by-betting framework that leverages predictions on unlabeled data to enhance the power of sequential hypothesis testing. Given limited samples from the joint distribution of

(X,Y)

, and additional unlabeled samples from the marginal of

X

, we ask how unlabeled data can be used to hypothesize about the distribution of

Y

, and the conditional distribution of

Y\mid X

. We introduce an e-statistic and use it to construct a sequential test. Under standard distributional assumptions -- label shift or concept shift -- we establish that the test is anytime valid. Furthermore, we show that for binary data, the e-statistic has non-trivial power. Crucially, our approach retains these properties even when the underlying predictions are inaccurate. Through simulations and applications to large language models evaluation, we demonstrate power gains over baseline approaches, including prediction-powered inference. These gains persist even with relatively limited unlabeled data and when predictions have low accuracy due to weak correlation between

X

and

Y

Yaniv Tenzer, Elad Tolochinsky, Yaniv Romano

Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology

Unlabeled Data Two-Sample Testing Model Evaluation Posterior Predictive Distribution Conditional Distribution

Ask PDF