Organizations adopting AI face the challenge of deciding which outputs can be released without human review while meeting the quality standards of human experts. We develop a statistical framework for delegating tasks between experts and black-box generate-verify workflows, in which AI-generated answers are verified and revised over multiple rounds. Before deployment, human experts evaluate selected outputs, and their judgments are used to calibrate the verifier threshold and number of attempts. Our method selects the policy with the highest estimated automation rate among statistically approved candidates, without modifying the generator or verifier. We establish finite-sample control of the error rate among released outputs, characterize the expert evaluations sufficient to attain a target automation rate, and derive a minimax lower bound establishing rate optimality in the quality margin and required confidence. In a MultiHiertt financial question-answering case study, average held-out automation increases from 2.50% to 24.17% as expert evaluations accumulate, while released-output error rates remain around 1.20%. In contrast, heuristics that release answers after consistently high verifier scores either violate the 5% error tolerance or automate at most 1.30% of tasks. Synthetic experiments further examine these theoretical predictions under controlled settings.
Figures & tables
Figure 1: Generate–verify workflow, adapted from the workflow in Anthropic (2024) . A generator proposes a candidate, and a verifier assigns a score and provides feedback. The generator may revise the answer in another round if it does not pass verification.
Figure 2: Overview of the proposed human–AI collaboration framework. Before deployment, the organization runs the generate–verify workflow on pilot tasks and collects human expert evaluations. These evaluations and the workflow execution records guide the selection of a verifier threshold and round limit. During deployment, the AI releases the first output that passes the selected threshold within the allowed attempts; otherwise, the task is handed to a human expert.
Round t
1
2
3
4
5
6
7
8
9
10
Accuracy (%)
53.60
78.31
83.39
85.70
86.87
87.54
87.99
88.26
88.53
88.65
Table 1: Answer accuracy across generation rounds.
Method
n
Automation (SD)
Released error
5% requirement
Proposed
1,000
2.50% (7.63)
1.20%
Meets
Proposed
1,500
6.45% (10.94)
1.19%
Meets
Proposed
2,000
11.45% (12.50)
1.19%
Meets
Proposed
2,500
15.00% (12.46)
1.20%
Meets
Proposed
3,000
17.50% (11.65)
1.20%
Meets
Proposed
4,000
21.67% (8.64)
1.20%
Meets
Table 2: Held-out deployment performance of the proposed method and verifier-based heuristics.
Policy (s,m)
Calibration automation
Calibration error
(0.90,1)
31.71%
13.46%
(0.90,2)
62.87%
14.44%
(0.95,1)
29.40%
12.21%
(1.00,1)
2.17%
6.25%
(1.00,10)
18.29%
1.48%
Table 3: Automation and released-output quality of representative policies on challenging financial questions.
α
n
Selected policies (out of 30)
Mean held-out automation (SD)
Held-out error
0.05
200
πesc 30
0.00% (0.00)
—
0.05
400
πesc 30
0.00% (0.00)
—
0.05
600
πesc 30
0.00% (0.00)
—
0.05
800
πesc 30
0.00% (0.00)
—
0.10
200
πesc 30
0.00% (0.00)
—
0.10
400
πesc 30
0.00% (0.00)
—
Table 4: Expert evaluation budgets and deployment performance under two quality requirements.
Figure 3: Effect of the quality margin on expert evaluation requirements. Panel (a) shows the probability of achieving the target automation rate as the evaluation budget increases. Panel (b) shows the minimum evaluation budget required to achieve the target with probability at least 90%, plotted against 1/Δe2 .
Figure 4: Effect of automation-rate separation on expert evaluation requirements. Panel (a) shows the probability of achieving the target automation rate as the evaluation budget increases. Panel (b) shows the minimum evaluation budget required to achieve the target with probability at least 90%, plotted against 1/Δr2 .
LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but it also creates a release decision: when to stop and output the current result? This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored, yet the likelihood models or exchangeability assumptions typically used for calibration are unavailable. We propose an always-valid release wrapper for existing generator-evaluator pipelines. The wrapper builds a hard-negative reference pool of high-scoring failures, calibrates deployment-time evaluator scores against this pool, and accumulates the resulting evidence with an e-process. This separates two roles: the reference pool turns black-box scores into conservative evidence, while the e-process provides validity under optional stopping. In theory, we show that a conservative reference pool yields finite-sample control of the probability of releasing on infeasible tasks, that is, tasks for which the given workflow is not capable of producing a reliable solution. We also characterize conditions under which the same conservative rule still achieves nontrivial release on feasible tasks. In an MBPP+ coding-agent case study, the wrapper reduces premature incorrect release relative to baseline stopping rules while still releasing on tasks for which the workflow repeatedly accumulates moderate supporting evidence.
Young Hyun Cho, Will Wei Sun
Department of Statistics, Purdue University. · Mitch Daniels School of Business, Purdue University.
As AI systems enter institutional workflows, workers must decide whether to delegate task execution to AI and how much effort to invest in verifying AI outputs, while institutions evaluate workers using outcome-based standards that may misalign with workers' private costs. We model delegation and verification as the solution to a rational worker's optimization problem, and define worker quality by evaluating an institution-centered utility (distinct from the worker's objective) at the resulting optimal action. We formally characterize optimal worker workflows and show that AI induces phase transitions, where arbitrarily small differences in verification ability lead to sharply different behaviors. As a result, AI can amplify workers with strong verification reliability while degrading institutional worker quality for others who rationally over-delegate and reduce oversight, even when baseline task success improves and no behavioral biases are present. These results identify a structural mechanism by which AI reshapes institutional worker quality and amplifies quality disparities between workers with different verification reliability.
Human feedback is critical for aligning AI systems to human values. As AI capabilities improve and AI is used to tackle more challenging tasks, verifying quality and safety becomes increasingly challenging. This paper explores how we can leverage AI to improve the quality of human oversight. We focus on an important safety problem that is already challenging for humans: fact-verification of AI outputs. We find that combining AI ratings and human ratings based on AI rater confidence is better than relying on either alone. Giving humans an AI fact-verification assistant further improves their accuracy, but the type of assistance matters. Displaying AI explanation, confidence, and labels leads to over-reliance, but just showing search results and evidence fosters more appropriate trust. These results have implications for Amplified Oversight -- the challenge of combining humans and AI to supervise AI systems even as they surpass human expert performance.
Rishub Jain, Sophie Bridgers, Lili Janzer +3
1Google DeepMind · 2Work done while previously at Google DeepMind