OmniMechanism Design for Human-AI Collaboration with Impressionable Minds

Tianwei Gao; Yifei Pang; Lunjia Hu; Kate Donahue; Lihua Lei; Vahid Tarokh; Zhun Deng

cs.AIOct 30, 2026

OmniMechanism Design for Human-AI Collaboration with Impressionable Minds

Authors: Tianwei Gao, Yifei Pang, Lunjia Hu, Kate Donahue, Lihua Lei, Vahid Tarokh, Zhun Deng

Organizations: University of North Carolina at Chapel Hill · Northeastern University · University of Illinois at Urbana-Champaign · Stanford University ·

Abstract

As AI systems are increasingly deployed in high-stakes domains such as healthcare, effectively integrating human expertise with AI outputs is critical for reliable decision-making. A substantial body of theoretical work studies optimal mechanisms for human-AI collaboration (HAC), focusing on when to involve humans and how to combine their expertise with AI predictions. However, these approaches largely overlook a key factor highlighted by empirical evidence: human judgments are impressionable and can vary depending on how AI outputs are presented. For example, when AI systems defer decisions to humans, revealing or withholding the AI prediction can significantly influence human judgments. Moreover, existing work neglects deployment-time variability, where conditions, particularly the cost of eliciting human input, may differ substantially from those observed during training. To address these challenges, we introduce a rigorous and theoretically grounded framework for human-AI collaboration, termed collaborative omnimechanism design, based on a novel and computationally efficient contextual bandit variant of boosting for outcome indistinguishability. In particular, we reduce a calibration-type outcome indistinguishability condition to weighted calibration, providing a novel time-efficiency result, bypassing the high-dimension curse induced by complicated collaboration methods and outcome space. Our framework provides a principled characterization of human impressionability and introduces a unified optimization scheme that enables robust and adaptable decision-making across diverse query costs and evaluation objectives without retraining. We evaluate the framework in three HAC settings, examining adaptation to query costs and prediction objectives without retraining, and show that its induced policies perform comparably to or better than the evaluated baselines.

Figures & tables

Figure 1: OmniMechanism setup and workflow diagram.

p^{(t+1)}(o\mid z,a)=\dfrac{p^{(t)}(o\mid z,a)\exp(\eta g^{(t)}(z,a,o))}{\sumop\displaylimits_{o^{\prime}\in\mathcal{O}}p^{(t)}(o^{\prime}\mid z,a)\exp(\eta g^{(t)}(z,a,o^{\prime}))}.

Algorithm 1 COI-Boosting Algorithm

Notation	Description
$x\in\mathcal{X},\ z=(x,y_{M})\in\mathcal{Z}$	Input and observed context; $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$ .
$M;\ y_{M},y_{H},y_{\mathrm{final}}\in\mathcal{Y}$	Black-box AI model; AI, human, and final predictions.
$y_{\mathrm{true}}\in\mathcal{Y}_{\mathrm{true}}$	Ground-truth label and its space.
$r\in\mathcal{R},\ \phi\in\Phi,\ g\in\mathcal{G}$	Deferral, signal-presentation, and collaboration rules; $s=\phi(z)\in\mathcal{S}$ is the displayed signal.
$a=(r,s,g)\in\mathcal{A}$	Collaboration action; $\mathcal{A}=\{0,1\}\times\mathcal{S}\times\mathcal{G}$ . Here $r$ is a binary decision.
$o\in\mathcal{O}$	Outcome $o=(y_{\mathrm{final}},y_{\mathrm{true}})$ ; $\mathcal{O}=\mathcal{Y}\times\mathcal{Y}_{\mathrm{true}}$ .

Table 1: Main notation for collaborative omnimechanism design.

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.768	0.768	1.000	0.722	0.722	1.000
	Manipulation	0.776	0.776	1.000	0.733	0.733	1.000
	Global Best	0.777	0.777	0.000	0.792	0.792	0.227
	Ours (Boosting)	0.778	0.778	0.178	0.785	0.785	0.378
	Ours (NN)	0.777	0.777	0.043	0.806	0.806	0.099

Table 2: Policy performance under human-involvement costs

c\in\{0,0.1,0.2\}

. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underlined , respectively.

Panel of Figure 2 — Figure 2: Training dynamics of the boosting-based omnimechanism over boosting rounds.

		Ours (NN)			Global Best		$\Delta$ Util.
$c$	Pathology	Score	Hum.	Util.	Action	Util.	Ours $-$ Global
0.00	Airspace Opacity	0.795	0.938	0.795	AI+H	0.771	$+0.024$
0.00	Cardiomediastinal Abnorm	0.768	0.785	0.768	AI+H	0.743	$+0.025$
0.02	Airspace Opacity	0.774	0.538	0.763	AI+H	0.751	$+0.012$
0.02	Cardiomediastinal Abnorm	0.767	0.015	0.767	AI+H	0.723	$+0.044$
0.05	Airspace Opacity	0.720	0.077	0.716	AI	0.697	$+0.019$

Table 3: Collab-CXR test performance by pathology. Score is

1-|\rho-y|

and

\mathrm{Util.}=\mathrm{Score}-c\cdot\mathrm{Hum.}

. Global Best is selected overall on training data, not separately by pathology.

	Ours (NN)			Global Best		$\Delta$ Loss
$c$	Error	Rev.	Total	Action	Total	Global $-$ Ours
0.000	0.1053	0.0278	0.1053	$\textsc{AI}_{2}$	0.1335	$+0.0281$
0.010	0.1052	0.0163	0.1054	$\textsc{AI}_{2}$	0.1335	$+0.0281$
0.025	0.1054	0.0115	0.1057	$\textsc{AI}_{2}$	0.1335	$+0.0278$
0.050	0.1054	0.0077	0.1057	$\textsc{AI}_{2}$	0.1335	$+0.0277$
0.100	0.1056	0.0000	0.1056	$\textsc{AI}_{2}$	0.1335	$+0.0279$

Table 4: AMES simulator test performance on 1,044 contexts from 87 held-out participants (seed 42). Error is continuous APE, Rev. is the fraction of contexts requesting an additional human revision, and

\mathrm{Total}=\mathrm{Error}+c\cdot\mathrm{Rev.}

. Positive

\Delta

Loss indicates lower total loss for Ours. Global Best is selected separately for each cost on training data.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.695	0.695	1.000	0.699	0.699	1.000
	Manipulation	0.697	0.697	1.000	0.710	0.710	1.000
	Global Best	0.702	0.702	0.182	0.743	0.743	0.532
	Ours (boosting)	0.698	0.698	0.709	0.757	0.757	0.450
	Ours (NN)	0.698	0.698	0.001	0.745	0.745	0.157

Appendix

Table 5: Policy performance after shifting AI advice to approximately

70\%

accuracy. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underline , respectively.

		Art			Sarcasm
$c$	Method	Acc.	Util.	Hum.	Acc.	Util.	Hum.
0.0	No Action	0.618	0.618	1.000	0.673	0.673	1.000
	Manipulation	0.618	0.618	1.000	0.671	0.671	1.000
	Global Best	0.615	0.615	0.409	0.681	0.681	0.797
	Ours (boosting)	0.617	0.617	0.602	0.683	0.683	0.752
	Ours (NN)	0.616	0.616	0.273	0.677	0.677	0.336

Appendix

Table 6: Policy performance after shifting AI advice to approximately

60\%

accuracy. We report accuracy (Acc.), utility (

\mathrm{Util.}=\mathrm{Acc.}-c\cdot\mathrm{Hum.}

), and human-involvement rate (Hum.). The best and second-best utilities in each setting are shown in bold and underline , respectively.

$c$	Pathology	AI	H	AI+H	Hum.
0.00	Airspace Opacity	4	39	22	0.938
0.00	Cardiomediastinal Abnorm.	14	4	47	0.785
0.02	Airspace Opacity	30	29	6	0.538
0.02	Cardiomediastinal Abnorm.	64	0	1	0.015
0.05	Airspace Opacity	60	5	0	0.077
0.05	Cardiomediastinal Abnorm.	65	0	0	0.000

Appendix

Table 7: Actions selected by the post-processed policy on the Collab-CXR test set. Each pathology contains 65 test X-rays.

$c$	NN Sim.	NN Logged	GB
0.000	0.1053	0.1061	0.1335
0.005	0.1054	0.1065	0.1335
0.010	0.1054	0.1055	0.1335
0.025	0.1057	0.1058	0.1335
0.050	0.1057	0.1066	0.1335
0.075	0.1057	0.1065	0.1335

Appendix

Table 8: AMES evaluation loss (APE plus marginal revision cost) on 87 held-out participants and 1,044 paired judgments. Sim. uses simulator rollouts; Logged uses Eq. 12 . Global Best (GB) selects

\textsc{AI}_{2}

, so its simulated and logged losses coincide. Lower values are better.

$c$	$\textsc{H}_{0}$	$\textsc{AI}_{1}$	$\textsc{AI}_{2}$	Rev.	Matched
0.000	67	473	475	29	5
0.005	69	473	475	27	5
0.010	69	477	481	17	3
0.025	70	478	484	12	2
0.050	70	480	486	8	2
0.075	70	481	491	2	1

Appendix

Table 9: Actions selected by the AMES policy on the 1,044 evaluation contexts. Rev. aggregates the six revision actions. Matched counts selected revisions whose action equals the logged condition; it is a subset of Rev.

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 458 papers

Mar 19, 2026

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.

Min Hun Lee

Singapore Management University, Singapore

Human-AI Decision Making Human-AI Collaboration Human-in-the-Loop Evaluation

Ask PDF

Jul 3, 2026

Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

The use of Large Language Models (LLMs) across diverse areas of human activity-ranging from everyday tasks to safety-critical applications-aims to enhance decision-making effectiveness with minimal human feedback. Concurrently, it seeks to align decisions with human expectations, preferences, and needs while mitigating risks associated with AI non-determinism. However, humans frequently over- or under-rely on AI recommendations, and current AI systems remain poorly calibrated to human expectations. To address these challenges, we introduce a human-AI collaborative decision-making framework designed to augment human capabilities and align AI agents with human preferences and expectations. Specifically, this paper (a) formulates the collaborative decision-making task as a stochastic game between an AI agent and a human player, and (b) proposes the Human-Centric Reflective Architecture (HCRA), which integrates human-calibrated models with reinforcement learning agents that leverage linguistic feedback in an iterative, reflective process. Evaluation results demonstrate that HCRA enhances decision-making effectiveness and delivers high-quality recommendations.

Andreas Kouridakis, Dimitrios Patiniotis Spyropoulos, George Vouros

University of Piraeus Piraeus, Greece

Human Preference Modeling Human-in-the-Loop AI Human-AI Decision Making Human-AI Collaboration RL from Human Feedback

Ask PDF

Feb 23, 2026

Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration

In human-AI decision making, designing AI that complements human expertise has been a natural strategy to enhance human-AI collaboration, yet it often comes at the cost of decreased AI performance in areas of human strengths. This can inadvertently erode human trust and cause them to ignore AI advice precisely when it is most needed. Conversely, an aligned AI fosters trust yet risks reinforcing suboptimal human behavior and lowering human-AI team performance. In this paper, we start by identifying this fundamental tension between performance-boosting (i.e., complementarity) and trust-building (i.e., alignment) as an inherent limitation of the traditional approach for training a single AI model to assist human decision making. To overcome this, we introduce a novel human-centered adaptive AI ensemble that strategically toggles between two specialist AI models - the aligned model and the complementary model - based on contextual cues, using an elegantly simple yet provably near-optimal Rational Routing Shortcut mechanism. Comprehensive theoretical analyses elucidate why the adaptive AI ensemble is effective and when it yields maximum benefits. Moreover, experiments on both simulated and real-world data show that when humans are assisted by the adaptive AI ensemble in decision making, they can achieve significantly higher performance than when they are assisted by single AI models that are trained to either optimize for their independent performance or even the human-AI team performance.

Hasan Amin, Ming Yin, Rajiv Khanna

Purdue University, USA

Human-AI Complementarity Human-AI Collaboration Human-AI Decision Making Ensemble Learning

Ask PDF

Abstract

Human-AI Collaboration

Explore similar work

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration