Toward Human-AI Complementarity Across Diverse Tasks

Yuzheng Xu; Annya Dahmani; Matthew D. Blanchard; Niclas Dern; Edy Nastase; Francesca Bianco; Maja Pavlovic; Sukanya Krishna

cs.HCApr 13, 2026

Toward Human-AI Complementarity Across Diverse Tasks

Authors: Yuzheng Xu, Annya Dahmani, Matthew D. Blanchard, Niclas Dern, Edy Nastase, Francesca Bianco, Maja Pavlovic, Sukanya Krishna, +8 more

Organizations: The University of Tokyo · UC Berkeley · University of Sydney · Queen Mary University London · Harvard University · UIUC · NIT Agartala · University of Pennsylvania · Princeton University

Abstract

Human-AI complementarity, the idea that combining human and AI judgments can outperform either alone, offers a promising pathway toward robust oversight of advanced AI systems. However, whether human-AI complementarity can be achieved on realistic tasks remains an open question. We investigate this through two approaches: hybridization and two AI assistance methods (top-2 assistance and subtask delegation), evaluated on a multi-domain dataset of 1,886 samples spanning knowledge, factuality, long-context reasoning, and deception detection. We find only modest complementarity gains. Baseline hybridization yields just +0.4 percentage points (pp) over AI alone (69.3% vs 68.9%), limited both by a small complementarity region (only 8.9% of items where AI errs but humans do not) and the inability of confidence-based routing to identify it, since the model's confidence is similarly distributed across correct and incorrect predictions. Applied when AI has low confidence, top-2 assistance increases human accuracy from 28.4% to 38.3%, surpassing AI alone (37.7%) -- but primarily because humans adopt correct AI suggestions, not because they successfully override AI errors. These findings suggest that the primary bottleneck is not human task accuracy per se, but the ability to route decisions to humans when it matters and to design assistance methods that enable humans to catch AI mistakes. Our quantitative and qualitative analyses pinpoint where and why each method succeeds or fails, offering concrete targets for future work. We will release our dataset and code upon request to support progress toward more effective human-AI collaboration for AI oversight.

Figures & tables

Figure 1: Overview of our experimental pipeline. Each item is independently evaluated by AI (majority vote over 20 GPT-5-mini responses OpenAI, 2025 ) and human raters under three conditions: unassisted, with top-2 assistance, and with subtask delegation. Both AI and human confidence scores are calibrated via isotonic regression and used for confidence-based hybridization routing, which selects the AI or human answer per item based on calibrated confidence thresholds. Section numbers in parentheses indicate where each component is described.

Dataset	Type	N (test)	Human Maj	Human Ind	AI Acc
FACTS Search	FT	180	69.4%	58.5%	86.7%
QuALITY	MC	180	61.1%	51.5%	80.6%
Big-Bench	MC/FT	232	29.7%	24.0%	61.6%
GPQA Diamond	MC	88	44.3%	38.3%	68.2%
Hidden Agenda	MC	60	83.3%	77.8%	81.7%
Humanity’s Last Exam	MC/FT	88	12.5%	9.5%	19.3%

Table 1: Baseline accuracy comparison. Human and AI (GPT-5-mini) accuracy on the test set (N=952 excluding SimpleQA). SimpleQA was excluded from hybridization experiments due to high AI accuracy (98.3%), leaving insufficient headroom for improvement.

	Human	AI	Baseline (best)	Top-2 (best)	Subtask (best)
Maj.	49.9%	68.9%	69.3% (2T/1TC)	69.3% (1TC)	69.2% (1T)
vs AI	$-$ 19.0pp	—	+0.4pp	+0.4pp	+0.3pp
Ind.	43.5%	68.9%	69.2% (2T)	69.1% (1T/1TC)	69.2% (1T)
vs AI	$-$ 25.4pp	—	+0.3pp	+0.2pp	+0.3pp

Table 2: Hybridization results on the full test set (N=952). Best routing method per condition, selected from 1-threshold (1T), 2-threshold (2T), and 1-threshold+compare (1TC).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Confidence Method	Calibration	ECE $\downarrow$	Brier $\downarrow$	Accuracy
ai_direct	Original	0.182	0.211	68.9%
ai_direct	Platt	0.048	0.179	68.9%
ai_direct	Isotonic	0.077	0.174	68.9%
ai_computed (MC)	Original	0.182	0.218	71.0%
ai_computed (MC)	Platt	0.065	0.190	71.0%
ai_computed (MC)	Isotonic	0.080	0.191	71.0%

Appendix

Table 3: Calibration metrics for AI confidence methods (40/60 split, evaluated on test set). ECE = Expected Calibration Error, Brier = Brier Score. Lower is better for both metrics.

Majority vote
Dataset	N	Human	AI	Base 1T	Base 2T	Base 1TC	Top2 1T	Top2 2T	Top2 1TC	Sub 1T	Sub 2T	Sub 1TC
FACTS	180	69.4%	86.7%	88.3%	88.3%	88.3%	86.7%	86.7%	87.2%	87.2%	87.2%	87.2%
QuALITY	180	61.1%	80.6%	80.6%	80.6%	80.6%	80.6%	80.6%	80.6%	80.6%	80.6%	80.6%
bbeh	232	29.7%	61.6%	61.6%	61.6%	61.6%	61.6%	61.2%	62.1%	62.9%	62.9%	62.5%
gpqa	88	44.3%	68.2%	68.2%	69.3%	69.3%	70.5%	70.5%	70.5%	65.9%	65.9%	64.8%
Hidden Agenda	60	83.3%	81.7%	81.7%	81.7%	81.7%	81.7%	83.3%	81.7%	81.7%	81.7%	81.7%

Appendix

Table 4: Per-dataset hybridization results (40/60 split). Top panel: majority vote (N=952 items). Bottom panel: individual responses (N=2856 observations). 1TC = 1-threshold + compare.

Majority vote
Dataset	N	Human	T2 Raw	AI †	B 1T	B 2T	B 1TC	T2 1T	T2 2T	T2 1TC	S 1T	S 2T	S 1TC
FACTS	20	45.0%	30.0%	30.0%	45.0%	45.0%	45.0%	30.0%	30.0%	35.0%	35.0%	35.0%	36.8%
QuALITY	16	37.5%	56.2%	62.5%	62.5%	62.5%	62.5%	62.5%	62.5%	62.5%	62.5%	62.5%	71.4%
bbeh	46	28.3%	23.9%	26.1%	26.1%	26.1%	26.1%	26.1%	23.9%	28.3%	32.6%	32.6%	30.2%
gpqa	12	50.0%	50.0%	33.3%	33.3%	33.3%	50.0%	50.0%	50.0%	50.0%	16.7%	16.7%	8.3%
Hidden Agenda	7	57.1%	100%	85.7%	85.7%	57.1%	85.7%	85.7%	100%	85.7%	85.7%	85.7%	85.7%

Appendix

Table 5: Per-dataset hybridization results on the 122-item low-confidence test subset. 1TC = 1-threshold + compare. AI alone uses majority vote over 20 sampled responses. T2 Raw = top-2 assisted human accuracy without hybridization. B = baseline, T2 = top-2, S = subtask. 1T = 1-threshold, 2T = 2-threshold, 1TC = 1-threshold + compare. Per-dataset N is small (4–46 majority); interpret with caution.

Majority vote
	Raw Accuracy					Oracle (perfect routing)
Dataset	N	Baseline	Top-2	Deleg	AI	O(Base)	O(Top-2)	O(Deleg)
FACTS	20	45.0%	30.0%	55.0%	30.0%	60.0%	40.0%	65.0%
QuALITY	16	37.5%	56.2%	62.5%	62.5%	75.0%	87.5%	75.0%
bbeh	46	28.3%	23.9%	26.1%	26.1%	37.0%	30.4%	39.1%
gpqa	12	50.0%	50.0%	8.3%	33.3%	75.0%	66.7%	41.7%

Appendix

Table 6: Complementarity ceiling analysis on the 122-item low-confidence test subset. O(X) = max(X, AI) per item, the accuracy achievable with perfect routing between X and AI.

	Individual Responses				Majority Responses
Predictor	OR	95% CI Low	95% CI Up	$p$	OR	95% CI Low	95% CI Up	$p$
Human Alone	0.27	0.05	1.38	0.12	0.35	0.04	3.09	0.35
AI Alone	1.03	0.73	1.45	0.89	1.31	0.63	2.73	0.47
Hybridized (Unassisted)	2.69	1.75	4.12	$<.001$ ***	1.71	0.83	3.55	0.15
Top-2 Assistance	2.80	1.74	4.49	$<.001$ ***	1.57	0.76	3.25	0.23
Delegation Assistance	2.82	1.80	4.42	$<.001$ ***	1.57	0.76	3.25	0.23

Appendix

Table 7: Overall mixed-effects model results on the 122-item test subset.

Contrast	OR	SE	95% CI Low	95% CI Up	$p$
AI Alone vs. Human Alone	1.03	0.18	0.63	1.68	1.00
Hybridized vs. Human Alone	2.69	0.59	1.45	4.96	$<.001$ ***
Hybridized vs. AI Alone	2.62	0.44	1.63	4.21	$<.001$ ***
Top-2 vs. Human Alone	2.80	0.68	1.42	5.51	$<.001$ ***
Top-2 vs. AI Alone	2.72	0.51	1.60	4.63	$<.001$ ***
Top-2 vs. Hybridized	1.04	0.24	0.54	2.00	1.00

Appendix

Table 8: Pairwise contrasts using individual-level data (Holm-adjusted).

Figure 2: Baseline human evaluation interface. Participants were shown task-specific instructions, the question prompt, and an input field for their answer (or multiple choice options). Participants provided their confidence using a slider ranging from 0 (“completely unsure”) to 100 (“completely certain”). A timer displayed the remaining time in the study.

Figure 3: Top-2 assistance interface. Participants were shown the task instructions and question prompt, along with the model’s top two candidate answers and brief explanations. Participants provided their own answer and a confidence rating.

Figure 4: Subtask delegation interface. The AI decomposes the original question into subtasks and proposes candidate answers with confidence estimates. Subtasks flagged as needing human input require a response, while high-confidence subtasks can be reviewed or supplemented with feedback.

Method	AUROC	ECE	Brier	Calibrated?
majority_vote_conf (isotonic)	0.809	0.034	0.173	Yes
majority_vote_conf (raw)	0.804	0.063	0.181	No
majority_vote_conf (temp. scaled)	0.804	0.049	0.182	Yes
majority_vote_conf (Platt)	0.804	0.078	0.180	Yes
majority_vote_conf (hist. binned)	0.804	0.081	0.175	Yes
mean_conf_majority	0.792	0.228	0.240	No

Appendix

Table 9: Top 10 confidence methods on the full benchmark (191 examples, sorted by AUROC).

Panel of Figure 5 — Figure 5: Reliability diagrams on the full benchmark. Left: majority_vote_conf and calibrated variants. Right: mean/median/extreme confidence methods. Calibrated variants collapse toward the diagonal.

Panel of Figure 6 — Figure 6: Confidence density histograms (correct vs. incorrect) on the full benchmark. majority_vote_conf shows clear separation between distributions.

Panel of Figure 7 — Figure 7: ECE (lower is better) and AUROC (higher is better) comparison across all methods on the full benchmark.

Method	AUROC	ECE	Brier	Calibrated?
cw_agreement (isotonic)	0.523	0.053	0.180	Yes
top2_gap	0.520	0.268	0.256	No
composite (isotonic)	0.516	0.052	0.181	Yes
composite (hist. binned)	0.515	0.069	0.182	Yes
prediction_entropy	0.515	—	—	No
agreement_rate	0.514	0.255	0.242	No

Appendix

Table 10: Top 10 confidence methods on safety subsets (306 examples, sorted by AUROC). All AUROC values are

\approx

0.50, indicating no discriminative power.

Figure 8: Per-dataset $2\times 2$ agreement matrices defined by majority-vote human correctness and AI correctness. Each cell shows the percentage and count of samples where both Human and AI are correct, only AI is correct, only humans are correct, or both are wrong.

Figure 9: Per-dataset AI confidence calibration curves. Points show predicted confidence vs. actual accuracy. Parenthetical counts after show the number of items observed.

Figure 10: Per-dataset AI confidence distributions, split by AI correctness. Green bars show confidence when the AI is correct; red bars when the AI is wrong. Parenthetical counts in each legend show the number of correct and incorrect items. Where green and red distributions overlap heavily, confidence cannot distinguish correct from incorrect predictions.

Figure 11: Human accuracy on the low-confidence subset (

N{=}122

), split by AI correctness. Upward triangles (

\blacktriangle

) = AI-correct items (46); downward triangles (

\blacktriangledown

) = AI-incorrect items (76). Dashed lines mark Baseline human accuracy in each group. Error bars show 95% bootstrap CIs.

AI Correctness	Condition	Obs.	Items	Accuracy [ 95% CI ]
Correct	Baseline (Unassisted)	138	46	43.5% [35.5, 51.4]
Correct	Top-2 Assistance	138	46	60.9% [52.9, 68.8]
Correct	Subtask Delegation	137	46	35.8% [27.7, 43.8]
Incorrect	Baseline (Unassisted)	228	76	19.3% [14.5, 24.6]
Incorrect	Top-2 Assistance	228	76	24.6% [19.3, 30.3]
Incorrect	Subtask Delegation	224	76	20.5% [15.6, 25.9]

Appendix

Table 11: Human accuracy by condition and AI correctness (low-confidence subset,

N{=}122

AI Correctness	Comparison	Diff (pp)	95% CI	$p$
Correct	Top-2 vs Baseline	$+$ 17.4	[ $+$ 5.8, $+$ 29.0]	0.006
Correct	Subtask vs Baseline	$-$ 7.7	[ $-$ 19.3, $+$ 3.9]	0.211
Correct	Top-2 vs Subtask	$+$ 25.1		$<$ 0.001
Incorrect	Top-2 vs Baseline	$+$ 5.3	[ $-$ 2.2, $+$ 12.7]	0.213
Incorrect	Subtask vs Baseline	$+$ 1.2	[ $-$ 6.2, $+$ 8.7]	0.810
Incorrect	Top-2 vs Subtask	$+$ 4.0		0.307

Appendix

Table 12: Pairwise accuracy differences vs. Baseline within each AI-correctness group (low-confidence subset,

N{=}122

p

-values from two-sided permutation tests (10,000 permutations).

	Inaccurate AI suggestions						Accurate AI suggestions
	Proper robustness			Over-reliance			Proper reliance			Under-reliance
Dataset	N	% bucket	% dataset	N	% bucket	% dataset	N	% bucket	% dataset	N	% bucket	% dataset
FACTS Search	5	27.8	16.7	8	53.3	26.7	1	4.2	3.3	0	0.0	0.0
QuALITY	0	0.0	0.0	1	6.7	3.3	10	41.7	33.3	7	35.0	23.3
BIG-Bench	7	38.9	9.6	5	33.3	6.9	5	20.8	6.9	4	20.0	5.5
GPQA diamond	1	5.6	7.1	0	0.0	0.0	2	8.3	14.3	3	15.0	21.4

Appendix

Table 13: Dataset breakdown across over- and under-reliance buckets (top-2 assistance condition). “% bucket” = share of that bucket from each dataset; “% dataset” = share of each dataset falling in that bucket.

Dataset	N items	Avg subtasks	Avg to human	Avg to AI	% to human
FACTS Search	30	4.87	2.37	2.50	48.6%
QuALITY	30	6.93	0.03	6.90	0.5%
Big-Bench	77	10.66	2.45	8.21	23.0%
GPQA Diamond	14	7.07	1.50	5.57	21.2%
Hidden Agenda	10	9.90	0.40	9.50	4.0%
Humanity’s Last Exam	14	10.07	3.29	6.79	32.6%

Appendix

Table 14: Subtask delegation pipeline: subtask routing breakdown by dataset (195 items across all study rounds including pilot). AI confidence threshold of 0.8 determines routing and subtasks below threshold are flagged for human input.

	GPQA Diamond	Other MC Datasets
Human Option 1 selection rate	38.2%	20.9%
Expected rate (uniform over 4 options)	25.0%	25.0%
Two-proportion $z$ -test	$z=7.89$ , $p<.001$

Appendix

Table 15: Option 1 selection rate on GPQA Diamond vs. other multiple-choice datasets.

Method	Accuracy	Category	Count	%
Human alone	49.9%	Both correct	390	41.0%
AI alone	68.9%	AI only correct	266	27.9%
Hybrid 2-threshold	69.3%	Human only correct	85	8.9%
Oracle (human OR AI)	77.8%	Neither correct	211	22.2%
Headroom (Oracle $-$ AI)	$+$ 8.9 pp
Captured (Hybrid $-$ AI)	$+$ 0.4 pp

Appendix

Table 16: Oracle upper bound and error decomposition (majority vote,

N{=}952

test items). The oracle answers correctly if either human or AI is correct. Headroom = Oracle

-

AI; Captured = Hybrid

-

AI; Capture rate = Captured / Headroom.

Category	Mean AI Confidence
Both correct	0.811
AI only correct	0.766
Human only correct	0.652
Neither correct	0.521

Appendix

Table 17: Mean calibrated AI confidence by agreement category (

N{=}952

). The “Human only correct” row reports the mean over all 85 items in that category; the 76 that remained with AI after routing had mean confidence 0.700.

Dataset	$N$	Human	AI	Hybrid	Oracle	Headroom	Captured
FACTS_search	180	69.4%	86.7%	88.3%	92.8%	$+$ 6.1	$+$ 1.7
QuALITY	180	61.1%	80.6%	80.6%	89.4%	$+$ 8.9	$+$ 0.0
Big-Bench	232	29.7%	61.6%	61.6%	68.5%	$+$ 6.9	$+$ 0.0
GPQA Diamond	88	44.3%	68.2%	69.3%	78.4%	$+$ 10.2	$+$ 1.1
Hidden Agenda	60	83.3%	81.7%	81.7%	93.3%	$+$ 11.7	$+$ 0.0
Humanity’s Last Exam	88	12.5%	19.3%	19.3%	25.0%	$+$ 5.7	$+$ 0.0

Appendix

Table 18: Per-dataset oracle analysis (majority vote,

N{=}952

). Headroom = Oracle

-

AI; Captured = Hybrid 2T

-

AI.

Explore similar work

Human-AI Complementarity: A Goal for Amplified Oversight
Oct 30, 2025Rishub Jain, Sophie Bridgers, Lili Janzer +3Claim Verification Trust in AI
1Google DeepMind · 2Work done while previously at Google DeepMind
When Can Human-AI Teams Outperform Individuals? Tight Bounds with Impossibility Guarantees
May 9, 2026Dongxin Guo, Jikun Wu, Siu-Ming YiuEnsemble Learning Human-AI Decision Making
The University of Hong Kong, Hong Kong, China · Stellaris AI Limited, Hong Kong, China · Brain Investing Limited, Hong Kong, China
Robust Human-AI Complementarity under Uncertainty
Jul 7, 2026Yewon Byun, Bryan WilderHuman-AI Decision Making Decision-Making under Uncertainty
Machine Learning Department, Carnegie Mellon University.

Oct 30, 2025J/K move · Enter open · S save

Human-AI Complementarity: A Goal for Amplified Oversight

Rishub Jain, Sophie Bridgers, Lili Janzer, Rory Greig+2

1Google DeepMind · 2Work done while previously at Google DeepMind

Claim Verification Trust in AI Human Oversight of AI

Open paper AskPDF arXiv

Human feedback is critical for aligning AI systems to human values. As AI capabilities improve and AI is used to tackle more challenging tasks, verifying quality and safety becomes increasingly challenging. This paper explores how we can leverage AI to improve the quality of human oversight. We focus on an important safety problem that is already challenging for humans: fact-verification of AI outputs. We find that combining AI ratings and human ratings based on AI rater confidence is better than relying on either alone. Giving humans an AI fact-verification assistant further improves their accuracy, but the type of assistance matters. Displaying AI explanation, confidence, and labels leads to over-reliance, but just showing search results and evidence fosters more appropriate trust. These results have implications for Amplified Oversight -- the challenge of combining humans and AI to supervise AI systems even as they surpass human expert performance.

Abstract

Explore similar work

Human-AI Complementarity: A Goal for Amplified Oversight

When Can Human-AI Teams Outperform Individuals? Tight Bounds with Impossibility Guarantees

Robust Human-AI Complementarity under Uncertainty