OmniMechanism Design for Human-AI Collaboration with Impressionable Minds

Tianwei Gao; Yifei Pang; Lunjia Hu; Kate Donahue; Lihua Lei; Vahid Tarokh; Zhun Deng

cs.AIOct 30, 2026

OmniMechanism Design for Human-AI Collaboration with Impressionable Minds

Authors: Tianwei Gao, Yifei Pang, Lunjia Hu, Kate Donahue, Lihua Lei, Vahid Tarokh, Zhun Deng

Organizations: University of North Carolina at Chapel Hill · Northeastern University · University of Illinois at Urbana-Champaign · Stanford University ·

Abstract

As AI systems are increasingly deployed in high-stakes domains such as healthcare, effectively integrating human expertise with AI outputs is critical for reliable decision-making. A substantial body of theoretical work studies optimal mechanisms for human-AI collaboration (HAC), focusing on when to involve humans and how to combine their expertise with AI predictions. However, these approaches largely overlook a key factor highlighted by empirical evidence: human judgments are impressionable and can vary depending on how AI outputs are presented. For example, when AI systems defer decisions to humans, revealing or withholding the AI prediction can significantly influence human judgments. Moreover, existing work neglects deployment-time variability, where conditions, particularly the cost of eliciting human input, may differ substantially from those observed during training. To address these challenges, we introduce a rigorous and theoretically grounded framework for human-AI collaboration, termed collaborative omnimechanism design, based on a novel and computationally efficient contextual bandit variant of boosting for outcome indistinguishability. In particular, we reduce a calibration-type outcome indistinguishability condition to weighted calibration, providing a novel time-efficiency result, bypassing the high-dimension curse induced by complicated collaboration methods and outcome space. Our framework provides a principled characterization of human impressionability and introduces a unified optimization scheme that enables robust and adaptable decision-making across diverse query costs and evaluation objectives without retraining. We evaluate the framework in three HAC settings, examining adaptation to query costs and prediction objectives without retraining, and show that its induced policies perform comparably to or better than the evaluated baselines.

Figures & tables

Figure 1: OmniMechanism setup and workflow diagram.

p^{(t+1)}(o\mid z,a)=\dfrac{p^{(t)}(o\mid z,a)\exp(\eta g^{(t)}(z,a,o))}{\sumop\displaylimits_{o^{\prime}\in\mathcal{O}}p^{(t)}(o^{\prime}\mid z,a)\exp(\eta g^{(t)}(z,a,o^{\prime}))}.

Algorithm 1 COI-Boosting Algorithm

Notation	Description
$x\in\mathcal{X},\ z=(x,y_{M})\in\mathcal{Z}$	Input and observed context; $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$ .
$M;\ y_{M},y_{H},y_{\mathrm{final}}\in\mathcal{Y}$	Black-box AI model; AI, human, and final predictions.
$y_{\mathrm{true}}\in\mathcal{Y}_{\mathrm{true}}$	Ground-truth label and its space.
$r\in\mathcal{R},\ \phi\in\Phi,\ g\in\mathcal{G}$	Deferral, signal-presentation, and collaboration rules; $s=\phi(z)\in\mathcal{S}$ is the displayed signal.
$a=(r,s,g)\in\mathcal{A}$	Collaboration action; $\mathcal{A}=\{0,1\}\times\mathcal{S}\times\mathcal{G}$ . Here $r$ is a binary decision.
$o\in\mathcal{O}$	Outcome $o=(y_{\mathrm{final}},y_{\mathrm{true}})$ ; $\mathcal{O}=\mathcal{Y}\times\mathcal{Y}_{\mathrm{true}}$ .

Table 1: Main notation for collaborative omnimechanism design.

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.768	0.768	1.000	0.722	0.722	1.000
	Manipulation	0.776	0.776	1.000	0.733	0.733	1.000
	Global Best	0.777	0.777	0.000	0.792	0.792	0.227
	Ours (Boosting)	0.778	0.778	0.178	0.785	0.785	0.378
	Ours (NN)	0.777	0.777	0.043	0.806	0.806	0.099

Table 2: Policy performance under human-involvement costs

c\in\{0,0.1,0.2\}

. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underlined , respectively.

Panel of Figure 2 — Figure 2: Training dynamics of the boosting-based omnimechanism over boosting rounds.

		Ours (NN)			Global Best		$\Delta$ Util.
$c$	Pathology	Score	Hum.	Util.	Action	Util.	Ours $-$ Global
0.00	Airspace Opacity	0.795	0.938	0.795	AI+H	0.771	$+0.024$
0.00	Cardiomediastinal Abnorm	0.768	0.785	0.768	AI+H	0.743	$+0.025$
0.02	Airspace Opacity	0.774	0.538	0.763	AI+H	0.751	$+0.012$
0.02	Cardiomediastinal Abnorm	0.767	0.015	0.767	AI+H	0.723	$+0.044$
0.05	Airspace Opacity	0.720	0.077	0.716	AI	0.697	$+0.019$

Table 3: Collab-CXR test performance by pathology. Score is

1-|\rho-y|

and

\mathrm{Util.}=\mathrm{Score}-c\cdot\mathrm{Hum.}

. Global Best is selected overall on training data, not separately by pathology.

	Ours (NN)			Global Best		$\Delta$ Loss
$c$	Error	Rev.	Total	Action	Total	Global $-$ Ours
0.000	0.1053	0.0278	0.1053	$\textsc{AI}_{2}$	0.1335	$+0.0281$
0.010	0.1052	0.0163	0.1054	$\textsc{AI}_{2}$	0.1335	$+0.0281$
0.025	0.1054	0.0115	0.1057	$\textsc{AI}_{2}$	0.1335	$+0.0278$
0.050	0.1054	0.0077	0.1057	$\textsc{AI}_{2}$	0.1335	$+0.0277$
0.100	0.1056	0.0000	0.1056	$\textsc{AI}_{2}$	0.1335	$+0.0279$

Table 4: AMES simulator test performance on 1,044 contexts from 87 held-out participants (seed 42). Error is continuous APE, Rev. is the fraction of contexts requesting an additional human revision, and

\mathrm{Total}=\mathrm{Error}+c\cdot\mathrm{Rev.}

. Positive

\Delta

Loss indicates lower total loss for Ours. Global Best is selected separately for each cost on training data.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.695	0.695	1.000	0.699	0.699	1.000
	Manipulation	0.697	0.697	1.000	0.710	0.710	1.000
	Global Best	0.702	0.702	0.182	0.743	0.743	0.532
	Ours (boosting)	0.698	0.698	0.709	0.757	0.757	0.450
	Ours (NN)	0.698	0.698	0.001	0.745	0.745	0.157

Appendix

Table 5: Policy performance after shifting AI advice to approximately

70\%

accuracy. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underline , respectively.

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.618	0.618	1.000	0.673	0.673	1.000
	Manipulation	0.618	0.618	1.000	0.671	0.671	1.000
	Global Best	0.615	0.615	0.409	0.681	0.681	0.797
	Ours (boosting)	0.617	0.617	0.602	0.683	0.683	0.752
	Ours (NN)	0.616	0.616	0.273	0.677	0.677	0.336

Appendix

Table 6: Policy performance after shifting AI advice to approximately

60\%

accuracy. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underline , respectively.

$c$	Pathology	AI	H	AI+H	Hum.
0.00	Airspace Opacity	4	39	22	0.938
0.00	Cardiomediastinal Abnorm.	14	4	47	0.785
0.02	Airspace Opacity	30	29	6	0.538
0.02	Cardiomediastinal Abnorm.	64	0	1	0.015
0.05	Airspace Opacity	60	5	0	0.077
0.05	Cardiomediastinal Abnorm.	65	0	0	0.000

Appendix

Table 7: Actions selected by the post-processed policy on the Collab-CXR test set. Each pathology contains 65 test X-rays.

$c$	NN Sim.	NN Logged	GB
0.000	0.1053	0.1061	0.1335
0.005	0.1054	0.1065	0.1335
0.010	0.1054	0.1055	0.1335
0.025	0.1057	0.1058	0.1335
0.050	0.1057	0.1066	0.1335
0.075	0.1057	0.1065	0.1335

Appendix

Table 8: AMES evaluation loss (APE plus marginal revision cost) on 87 held-out participants and 1,044 paired judgments. Sim. uses simulator rollouts; Logged uses Eq. 12 . Global Best (GB) selects

\textsc{AI}_{2}

, so its simulated and logged losses coincide. Lower values are better.

$c$	$\textsc{H}_{0}$	$\textsc{AI}_{1}$	$\textsc{AI}_{2}$	Rev.	Matched
0.000	67	473	475	29	5
0.005	69	473	475	27	5
0.010	69	477	481	17	3
0.025	70	478	484	12	2
0.050	70	480	486	8	2
0.075	70	481	491	2	1

Appendix

Table 9: Actions selected by the AMES policy on the 1,044 evaluation contexts. Rev. aggregates the six revision actions. Matched counts selected revisions whose action equals the logged condition; it is a subset of Rev.

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 458 papers

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making
Mar 19, 2026Min Hun LeeHuman-AI Decision Making Human-AI Collaboration
Singapore Management University, Singapore
Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making
Jul 3, 2026Andreas Kouridakis, Dimitrios Patiniotis Spyropoulos, George VourosHuman Preference Modeling Human-in-the-Loop AI
University of Piraeus Piraeus, Greece
Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration
Feb 23, 2026Hasan Amin, Ming Yin, Rajiv KhannaHuman-AI Complementarity Human-AI Collaboration
Purdue University, USA

Mar 19, 2026J/K move · Enter open · S save

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Min Hun Lee

Singapore Management University, Singapore

Human-AI Decision Making Human-AI Collaboration Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.

Abstract

Human-AI Collaboration

Explore similar work

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration