Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the human-AI workflow or replace it with the human alone or the agent alone. The human-AI workflow is worth keeping only if it beats both of those alternatives. Yet once it is deployed, neither alternative outcome is observed: recovering one means replaying the task under that alternative, and every replay costs expert time or compute. Under a fixed replay budget, the design question is therefore which tasks should be more likely to receive a human-only replay, and which an agent-only replay. Existing methods do not directly target this decision. Agent benchmarks do not choose which missing baseline to measure, variance-based sampling ignores which of the two comparisons is closer to failing, and Bayesian information methods focus on learning model parameters instead of making the deployment decision. We propose TEAM-Design, a rule that gives every task two replay probabilities, one per baseline. It raises a probability where the missing baseline outcome is hard to predict from what is already known about the task and where that comparison is harder to establish, and lowers it where replay is expensive. We prove that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the workflow beats both. We reanalyze 6 clinical settings, where no human-AI workflow beats both alternatives, and a coding benchmark, where one does, then evaluate TEAM-Design on synthetic designs and on a semi-synthetic design built from a real chest X-ray reader study. TEAM-Design works best when one of the two comparisons is clearly harder to settle than the other, and can do worse than variance-based allocation when the two are similarly difficult.
Figures & tables
Study
Reads
ΔH (vs. clinician)
ΔA (vs. AI)
Beats both
Chest X-ray, design 2, no history
1,980
+0.0056 (0.0026)
-0.0175 (0.0055)
no
Chest X-ray, design 2, with history
1,977
+0.0024 (0.0031)
-0.0143 (0.0063)
no
Chest X-ray, design 3, no history
875
+0.0185 (0.0073)
-0.0246 (0.0091)
no
Chest X-ray, design 3, with history
875
+0.0106 (0.0076)
-0.0157 (0.0073)
no
Melanoma, AI support
1,508
+0.0836 (0.0107)
-0.0206 (0.0108)
no
Melanoma, AI with explanations
1,508
+0.0713 (0.0107)
-0.0328 (0.0108)
no
Table 1: Matched human–AI reader studies judged against both baselines, with reader-clustered standard errors in parentheses. Bold marks a one-sided 95% lower bound above zero. In no setting does the assisted clinician beat both alternatives.
Figure 1: How the replay budget divides between the two baselines, aggregated from the task-level probabilities qiH and qiA . TEAM-Design raises the replay probabilities for the harder comparison when one clearly requires more evidence, and stays closer to an equal split otherwise.
Setting
TEAM-Design
Neyman
Uniform
Surrogate oracle
One harder comparison
77.1 (0.30)
65.1 (0.34)
65.8 (0.34)
79.8 (0.28)
Balanced comparisons
39.1 (0.35)
44.4 (0.35)
43.7 (0.35)
44.3 (0.35)
Unequal replay cost (5:1)
57.9 (0.35)
54.1 (0.35)
45.2 (0.35)
56.2 (0.35)
Boundary null
3.6 (0.13)
4.6 (0.15)
4.8 (0.15)
3.4 (0.13)
Boundary null, reversed pilot
3.7 (0.13)
4.5 (0.15)
4.7 (0.15)
3.3 (0.13)
Table 2: Beats-both decision rate: replications (%) with both lower bounds above zero; under the boundary null this is a false-claim rate at a nominal 5%. Monte Carlo errors in parentheses, R=20,000 per row. The uniform column is an equal cost split in the unequal-cost row. The oracle allocates from the true gains, a reference design and not a deployable rule.
Figure 2: Root mean squared error on the harder comparison as a function of the replay budget, on the structure of the chest X-ray study. All methods coincide at full replay.
Figure 3: Scope of TEAM-Design . (a) When the two comparisons are similarly difficult, TEAM-Design trails Neyman, with the gap shrinking as the pilot grows. (b) Performance against the stabilizer τ , with the pilot standard error marked.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Synthetic
Semi-synthetic
Gaussian
Pilot size
500
a quarter of cases
a tenth of the budget
Evaluation size
3,000
about 656
18,000
Replay budget B
0.4
0.15 to 1.0
—
Probability floor qj,min
0.02
0.05
—
Stabilizer τ
0.03
pilot standard error
0.03
Replications
20,000
200
30,000
Appendix
Table 3: Settings for the three constructed studies. The synthetic and semi-synthetic sizes count tasks and reads, and the Gaussian sizes count measurements.
Method
Power
False claim
Size-corrected
Share to harder
TEAM-Design
76.3
4.9
76.7
0.83
Bayesian decision-targeted, batch
75.2
5.0
75.0
0.77
Parameter-EIG
59.5
4.9
59.8
0.50
Equal split
60.1
5.0
60.3
0.50
Surrogate oracle
81.0
4.7
81.7
0.89
Appendix
Table 4: Gaussian acquisition at (ΔH,ΔA)=(0.02,0.10) with a shared pilot charged to a common budget. Every method spends the same total. False claim is measured at the least-favorable null ΔH=0 . Size-corrected power calibrates each critical value so all methods have exactly 5% false-claim rate; it needs the null distribution, so it upper-bounds any implementable correction. Every rule commits to an allocation before measuring, so every fixed-sample bound is exact. R=60,000 at the null and R=30,000 at the alternative.
Setting
TEAM-Design
Neyman
Uniform
Surrogate oracle
Replay cost ratio cH/cA
2:1
84.0 (0.26)
76.9 (0.30)
72.5 (0.32)
87.1 (0.24)
5:1
57.9 (0.35)
54.1 (0.35)
45.2 (0.35)
56.2 (0.35)
10:1
37.6 (0.34)
37.0 (0.34)
29.0 (0.32)
36.4 (0.34)
Outcome correlation ρ (Gaussian copula)
ρ=0.25 , bottleneck
80.5 (0.89)
65.8 (1.06)
66.9 (1.05)
82.2 (0.86)
Appendix
Table 5: Sensitivity sweeps. Beats-both decision rate with Monte Carlo standard errors in parentheses. The uniform column is an equal cost split in the unequal-cost rows. Cost-ratio rows use 20,000 replications and the 5:1 row repeats Table 2 , while the correlation and size sweeps use 2,000. The npilot=500 and neval=3,000 rows are the primary setting run once in each sweep. sweeps use 2,000.