Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the human-AI workflow or replace it with the human alone or the agent alone. The human-AI workflow is worth keeping only if it beats both of those alternatives. Yet once it is deployed, neither alternative outcome is observed: recovering one means replaying the task under that alternative, and every replay costs expert time or compute. Under a fixed replay budget, the design question is therefore which tasks should be more likely to receive a human-only replay, and which an agent-only replay. Existing methods do not directly target this decision. Agent benchmarks do not choose which missing baseline to measure, variance-based sampling ignores which of the two comparisons is closer to failing, and Bayesian information methods focus on learning model parameters instead of making the deployment decision. We propose TEAM-Design, a rule that gives every task two replay probabilities, one per baseline. It raises a probability where the missing baseline outcome is hard to predict from what is already known about the task and where that comparison is harder to establish, and lowers it where replay is expensive. We prove that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the workflow beats both. We reanalyze 6 clinical settings, where no human-AI workflow beats both alternatives, and a coding benchmark, where one does, then evaluate TEAM-Design on synthetic designs and on a semi-synthetic design built from a real chest X-ray reader study. TEAM-Design works best when one of the two comparisons is clearly harder to settle than the other, and can do worse than variance-based allocation when the two are similarly difficult.
Figures & tables
Study
Reads
ΔH (vs. clinician)
ΔA (vs. AI)
Beats both
Chest X-ray, design 2, no history
1,980
+0.0056 (0.0026)
-0.0175 (0.0055)
no
Chest X-ray, design 2, with history
1,977
+0.0024 (0.0031)
-0.0143 (0.0063)
no
Chest X-ray, design 3, no history
875
+0.0185 (0.0073)
-0.0246 (0.0091)
no
Chest X-ray, design 3, with history
875
+0.0106 (0.0076)
-0.0157 (0.0073)
no
Melanoma, AI support
1,508
+0.0836 (0.0107)
-0.0206 (0.0108)
no
Melanoma, AI with explanations
1,508
+0.0713 (0.0107)
-0.0328 (0.0108)
no
Table 1: Matched human–AI reader studies judged against both baselines, with reader-clustered standard errors in parentheses. Bold marks a one-sided 95% lower bound above zero. In no setting does the assisted clinician beat both alternatives.
Figure 1: How the replay budget divides between the two baselines, aggregated from the task-level probabilities qiH and qiA . TEAM-Design raises the replay probabilities for the harder comparison when one clearly requires more evidence, and stays closer to an equal split otherwise.
Setting
TEAM-Design
Neyman
Uniform
Surrogate oracle
One harder comparison
77.1 (0.30)
65.1 (0.34)
65.8 (0.34)
79.8 (0.28)
Balanced comparisons
39.1 (0.35)
44.4 (0.35)
43.7 (0.35)
44.3 (0.35)
Unequal replay cost (5:1)
57.9 (0.35)
54.1 (0.35)
45.2 (0.35)
56.2 (0.35)
Boundary null
3.6 (0.13)
4.6 (0.15)
4.8 (0.15)
3.4 (0.13)
Boundary null, reversed pilot
3.7 (0.13)
4.5 (0.15)
4.7 (0.15)
3.3 (0.13)
Table 2: Beats-both decision rate: replications (%) with both lower bounds above zero; under the boundary null this is a false-claim rate at a nominal 5%. Monte Carlo errors in parentheses, R=20,000 per row. The uniform column is an equal cost split in the unequal-cost row. The oracle allocates from the true gains, a reference design and not a deployable rule.
Figure 2: Root mean squared error on the harder comparison as a function of the replay budget, on the structure of the chest X-ray study. All methods coincide at full replay.
Figure 3: Scope of TEAM-Design . (a) When the two comparisons are similarly difficult, TEAM-Design trails Neyman, with the gap shrinking as the pilot grows. (b) Performance against the stabilizer τ , with the pilot standard error marked.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Synthetic
Semi-synthetic
Gaussian
Pilot size
500
a quarter of cases
a tenth of the budget
Evaluation size
3,000
about 656
18,000
Replay budget B
0.4
0.15 to 1.0
—
Probability floor qj,min
0.02
0.05
—
Stabilizer τ
0.03
pilot standard error
0.03
Replications
20,000
200
30,000
Appendix
Table 3: Settings for the three constructed studies. The synthetic and semi-synthetic sizes count tasks and reads, and the Gaussian sizes count measurements.
Method
Power
False claim
Size-corrected
Share to harder
TEAM-Design
76.3
4.9
76.7
0.83
Bayesian decision-targeted, batch
75.2
5.0
75.0
0.77
Parameter-EIG
59.5
4.9
59.8
0.50
Equal split
60.1
5.0
60.3
0.50
Surrogate oracle
81.0
4.7
81.7
0.89
Appendix
Table 4: Gaussian acquisition at (ΔH,ΔA)=(0.02,0.10) with a shared pilot charged to a common budget. Every method spends the same total. False claim is measured at the least-favorable null ΔH=0 . Size-corrected power calibrates each critical value so all methods have exactly 5% false-claim rate; it needs the null distribution, so it upper-bounds any implementable correction. Every rule commits to an allocation before measuring, so every fixed-sample bound is exact. R=60,000 at the null and R=30,000 at the alternative.
Setting
TEAM-Design
Neyman
Uniform
Surrogate oracle
Replay cost ratio cH/cA
2:1
84.0 (0.26)
76.9 (0.30)
72.5 (0.32)
87.1 (0.24)
5:1
57.9 (0.35)
54.1 (0.35)
45.2 (0.35)
56.2 (0.35)
10:1
37.6 (0.34)
37.0 (0.34)
29.0 (0.32)
36.4 (0.34)
Outcome correlation ρ (Gaussian copula)
ρ=0.25 , bottleneck
80.5 (0.89)
65.8 (1.06)
66.9 (1.05)
82.2 (0.86)
Appendix
Table 5: Sensitivity sweeps. Beats-both decision rate with Monte Carlo standard errors in parentheses. The uniform column is an equal cost split in the unequal-cost rows. Cost-ratio rows use 20,000 replications and the 5:1 row repeats Table 2 , while the correlation and size sweeps use 2,000. The npilot=500 and neval=3,000 rows are the primary setting run once in each sweep. sweeps use 2,000.
AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments can be biased or incomplete. We study selective human evaluation: given AI signals, which factors should be dispatched to human experts? We formulate this human--AI collaboration problem as a factor-level information-acquisition problem. Under squared loss, the optimal dispatching rule maximizes a contextual reward; under a linear model, this reward admits a closed-form decomposition into two interpretable terms: predictive relevance and residual uncertainty in the human evaluation after conditioning on the AI signal. We instantiate the framework on peer review, decomposing papers into ten aspects and evaluating up to 3,408 ICLR submissions across three LLMs and multiple regression heads. In this retrospective proxy, where human aspect signals are extracted from existing reviews, predictions that query three of the ten aspects under the optimal dispatching rules match the full ten-aspect human benchmark. The finding holds when the human signal comes from a single noisy reviewer, and across review years. These results position principled dispatching as a practical building block for scalable human--AI decision systems. We also discuss safeguards under which such a rule can serve as decision support that leaves the final decision with a human.
Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
Yuetai Li, Yichen Feng, Zhangchen Xu +21
University of Washington · 10Bake AI · University of California, Santa Barbara +10