Organizations adopting AI face the challenge of deciding which outputs can be released without human review while meeting the quality standards of human experts. We develop a statistical framework for delegating tasks between experts and black-box generate-verify workflows, in which AI-generated answers are verified and revised over multiple rounds. Before deployment, human experts evaluate selected outputs, and their judgments are used to calibrate the verifier threshold and number of attempts. Our method selects the policy with the highest estimated automation rate among statistically approved candidates, without modifying the generator or verifier. We establish finite-sample control of the error rate among released outputs, characterize the expert evaluations sufficient to attain a target automation rate, and derive a minimax lower bound establishing rate optimality in the quality margin and required confidence. In a MultiHiertt financial question-answering case study, average held-out automation increases from 2.50% to 24.17% as expert evaluations accumulate, while released-output error rates remain around 1.20%. In contrast, heuristics that release answers after consistently high verifier scores either violate the 5% error tolerance or automate at most 1.30% of tasks. Synthetic experiments further examine these theoretical predictions under controlled settings.
Figures & tables
Figure 1: Generate–verify workflow, adapted from the workflow in Anthropic (2024) . A generator proposes a candidate, and a verifier assigns a score and provides feedback. The generator may revise the answer in another round if it does not pass verification.
Figure 2: Overview of the proposed human–AI collaboration framework. Before deployment, the organization runs the generate–verify workflow on pilot tasks and collects human expert evaluations. These evaluations and the workflow execution records guide the selection of a verifier threshold and round limit. During deployment, the AI releases the first output that passes the selected threshold within the allowed attempts; otherwise, the task is handed to a human expert.
Round t
1
2
3
4
5
6
7
8
9
10
Accuracy (%)
53.60
78.31
83.39
85.70
86.87
87.54
87.99
88.26
88.53
88.65
Table 1: Answer accuracy across generation rounds.
Method
n
Automation (SD)
Released error
5% requirement
Proposed
1,000
2.50% (7.63)
1.20%
Meets
Proposed
1,500
6.45% (10.94)
1.19%
Meets
Proposed
2,000
11.45% (12.50)
1.19%
Meets
Proposed
2,500
15.00% (12.46)
1.20%
Meets
Proposed
3,000
17.50% (11.65)
1.20%
Meets
Proposed
4,000
21.67% (8.64)
1.20%
Meets
Table 2: Held-out deployment performance of the proposed method and verifier-based heuristics.
Policy (s,m)
Calibration automation
Calibration error
(0.90,1)
31.71%
13.46%
(0.90,2)
62.87%
14.44%
(0.95,1)
29.40%
12.21%
(1.00,1)
2.17%
6.25%
(1.00,10)
18.29%
1.48%
Table 3: Automation and released-output quality of representative policies on challenging financial questions.
α
n
Selected policies (out of 30)
Mean held-out automation (SD)
Held-out error
0.05
200
πesc 30
0.00% (0.00)
—
0.05
400
πesc 30
0.00% (0.00)
—
0.05
600
πesc 30
0.00% (0.00)
—
0.05
800
πesc 30
0.00% (0.00)
—
0.10
200
πesc 30
0.00% (0.00)
—
0.10
400
πesc 30
0.00% (0.00)
—
Table 4: Expert evaluation budgets and deployment performance under two quality requirements.
Figure 3: Effect of the quality margin on expert evaluation requirements. Panel (a) shows the probability of achieving the target automation rate as the evaluation budget increases. Panel (b) shows the minimum evaluation budget required to achieve the target with probability at least 90%, plotted against 1/Δe2 .
Figure 4: Effect of automation-rate separation on expert evaluation requirements. Panel (a) shows the probability of achieving the target automation rate as the evaluation budget increases. Panel (b) shows the minimum evaluation budget required to achieve the target with probability at least 90%, plotted against 1/Δr2 .