Trust Hinders Bidirectional Human-AI Alignment

Michaela Benk; Kacper Sokol

cs.AIOct 10, 2026

Trust Hinders Bidirectional Human-AI Alignment

Authors: Michaela Benk, Kacper Sokol

Organizations: Department of Informatics, University of Zurich, Switzerland · Department of Informatics, USI Lugano, Switzerland

Abstract

Trust calibration remains the dominant framing of AI-assisted decision making: the goal is to close the gap between perceived and actual trustworthiness of AI systems so that users rely on them appropriately. We argue that this approach, while useful for decision problems that are well-defined and clearly scoped, overly restricts what it means to make a decision with AI support. To overcome this limitation, we develop three dimensions of AI-assisted decision making. First, trusting AI is only one of several possible strategies for managing uncertainty. Second, trust calibration is often misapplied when AI models large-world decision problems, i.e., ill-defined situations where goals and states are only partially known or unknowable. Third, in such settings human-AI alignment requires addressing higher-order decision processes. We conclude by reflecting on the implications of our observations on the design of process-oriented human-AI coupled systems that jointly identify and negotiate uncertainty throughout the decision-making task.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Strategy	Description
Rules	Adopt a rule in anticipation of challenging or repetitive decisions; for example, “when it rains, always take the fastest route”. A rule is a full, or nearly full, ex ante specification of results in individual cases. Rules are error-prone in a characteristic way: they are over- and under-inclusive relative to the reasons that justify them.
Heuristics	Rely on a mental shortcut that bypasses case-by-case choices.
Presumptions	Act on the assumption that $P$ is true until circumstances $A$ arise, in which case stop relying on $P$ and reconsider the situation.
Standards	Adopt a vague, value-based criterion that structures judgment without settling it; for example, drive “responsibly”, act “reasonably” and convict only “beyond a reasonable doubt”.
Routines	Rely on a behavioural pattern that is habit-like but more voluntary and self-conscious than a habit; for example, lock the office door on every exit or take the same commute daily. Occasional error is accepted as the price of not deciding.
Small steps	Make a narrow, reversible decision now and defer answering the larger question.

Appendix

Table 1: Second-order strategies for managing uncertainty according to Sunstein and Ullmann-Margalit [20] . Note that relying on an AI recommendation is an instance of delegating – see the last entry in the table.

Human-AI Decision Making

Human-AI decision making studies how people and AI systems share, support, or delegate decisions across tasks and institutions.

See all 173 papers

Learning to Decide with AI Assistance under Human-Alignment
May 12, 2026Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-RodriguezAI-Assisted Decision Making Human-AI Decision Making
§Max Planck Institute for Biochemistry, Martinsried, Germany · †Max Planck Institute for Software Systems, Kaiserslautern, Germany
Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning
Jul 20, 2026Vijay Keswani, Breanna K. Nguyen, Cyrus Cousins +3Explainable Artificial Intelligence AI Alignment
Department of Computer Science and Engineering, IIT Delhi · Department of Philosophy, Duke University · Department of Computer Science, Carnegie Mellon University +1
Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration
Feb 23, 2026Hasan Amin, Ming Yin, Rajiv KhannaHuman-AI Complementarity Human-AI Collaboration
Purdue University, USA

May 12, 2026J/K move · Enter open · S save

Learning to Decide with AI Assistance under Human-Alignment

Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-Rodriguez

§Max Planck Institute for Biochemistry, Martinsried, Germany · †Max Planck Institute for Software Systems, Kaiserslautern, Germany

AI-Assisted Decision Making Human-AI Decision Making Statistical Learning Theory Online Learning

Open paper AskPDF arXiv

It is widely agreed that when AI models assist decision-makers in high-stakes domains by predicting an outcome of interest, they should communicate the confidence of their predictions. However, empirical evidence suggests that decision-makers often struggle to determine when to trust a prediction based solely on this communicated confidence. In this context, recent theoretical and empirical work suggests a positive correlation between the utility of AI-assisted decision-making and the degree of alignment between the AI confidence and the decision-makers' confidence in their own predictions. Crucially, these findings do not yet elucidate the extent to which this alignment influences the complexity of learning to make optimal decisions through repeated interactions. In this paper, we address this question in the canonical case of binary predictions and binary decisions. We first show that this problem is equivalent to a two-armed online contextual learning problem with full feedback, and establish a lower bound of

Ω(\sqrt{|H| \cdot |B| \cdot T} )

on the expected regret any learner can attain, where

H

and

B

denote the sets of human and AI confidence values. We then demonstrate that, under perfect alignment between AI and human confidence, a learner can attain an expected regret of

O(\sqrt{|H| \cdot T\log T})

and, when

\sqrt{|H|} = O(\log T)

and

B

is countable, a non-trivial generalization of the Dvoretzky-Kiefer-Wolfowitz inequality improves the regret bound to

O(\sqrt{T\log T})

. Taken together, these results reveal that alignment can reduce the complexity of learning to make decisions with AI assistance. Experiments on real data from two different human-subject studies where participants solve simple decision-making tasks assisted by AI models show that our theoretical results are robust to violations of perfect alignment.

Abstract

Human-AI Decision Making

Explore similar work

Learning to Decide with AI Assistance under Human-Alignment

Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning

Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaboration