Learning to Trust: How Humans Mentally Recalibrate AI Confidence Signals

ZhaoBin Li; Mark Steyvers

cs.HCMar 23, 2026

Learning to Trust: How Humans Mentally Recalibrate AI Confidence Signals

Authors: ZhaoBin Li, Mark Steyvers

Organizations: Department of Cognitive Sciences, University of California, Irvine

Abstract

Productive human-AI collaboration requires appropriate reliance, yet contemporary AI systems are often miscalibrated, exhibiting systematic overconfidence or underconfidence. We investigate whether humans can learn to mentally recalibrate AI confidence signals through repeated experience. In a behavioral experiment (N = 200), participants predicted the AI's correctness across four AI calibration conditions: standard, overconfidence, underconfidence, and a counterintuitive "reverse confidence" mapping. Results demonstrate robust learning across all conditions, with participants significantly improving their accuracy, discrimination, and calibration alignment over 50 trials. We present a computational model utilizing a linear-in-log-odds (LLO) transformation and a Rescorla-Wagner learning rule to explain these dynamics. The model reveals that humans adapt by updating their baseline trust and confidence sensitivity, using asymmetric learning rates to prioritize the most informative errors. While humans can compensate for monotonic miscalibration, we identify a significant boundary in the reverse confidence scenario, where a substantial proportion of participants struggled to override initial inductive biases. These findings provide a mechanistic account of how humans adapt their trust in AI confidence signals through experience.

Figures & tables

Figure 1: Experimental trial structure: participants viewed a 1-second colored dots animation, and then received the AI’s prediction on which color has the most dots and confidence score rounded to the nearest 10%. Participants then judge whether the AI was correct or wrong, and receive immediate feedback on their accuracy.

Figure 2: Probability densities of AI confidence distributions across the four conditions for correct and wrong decisions. The probabilities are rounded to the nearest 10% as shown to the experimental interface and normalized across both distributions.

Figure 3: Accuracy improvement across trial blocks for human participants and cognitive model in all four AI calibration conditions. Error bars represent 95% confidence intervals and the dashed line indicates 50% chance accuracy.

Figure 4: Hit rate increases and false alarm rate decreases across trial blocks for human participants and cognitive model in all four AI calibration conditions. Error bars represent 95% confidence intervals and the dashed line indicates 50% chance level.

Figure 5: Calibration curves showing AI accuracy (orange) and human perceived AI accuracy (blue dashed) across conditions. Left panel shows prior calibration at trial 1 from experiment-wide priors. Subsequent panels show posterior calibration at trial 50 from participant-level posteriors per experimental condition. Black diagonal dashed line shows perfect calibration. Shaded regions show 95% credible intervals.

Figure 6: Individual differences in the reverse-confidence condition. Left : Calibration curves at trial 50 showing AI accuracy (dashed green) and human perceived AI accuracy for learners (blue) and non-learners (orange). Right : Learning rates for confidence sensitivity (

w

) when the AI was correct (

\alpha_{w,g_{t}=1}

) and wrong (

\alpha_{w,g_{t}=0}

). Shaded regions and error bars represent 95% credible intervals.

Figure 7: Changes in baseline trust

b_{t}

(solid blue) and confidence sensitivity

w_{t}

(dashed orange) across trials in all four conditions. Shaded regions represent 95% credible intervals.

Explore similar work

Learning to Decide with AI Assistance under Human-Alignment
May 12, 2026Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-RodriguezAI-Assisted Decision Making Human-AI Decision Making
§Max Planck Institute for Biochemistry, Martinsried, Germany · †Max Planck Institute for Software Systems, Kaiserslautern, Germany
The Confidence Game: Strategic Miscalibration in Human-AI Delegation
Oct 7, 2026Raghu Arghal, Saswati Sarkar, Shirin Saeedi BidokhtiLLM Agent Reliability Confidence Estimation in Language Models
Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA
AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
May 27, 2026Maharshi Gor, Yoo Yeon Sung, Yu Hou +4Human-AI Agreement Trust in AI
University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA +1

May 12, 2026J/K move · Enter open · S save

Learning to Decide with AI Assistance under Human-Alignment

Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-Rodriguez

§Max Planck Institute for Biochemistry, Martinsried, Germany · †Max Planck Institute for Software Systems, Kaiserslautern, Germany

AI-Assisted Decision Making Human-AI Decision Making Statistical Learning Theory Online Learning

Open paper AskPDF arXiv

It is widely agreed that when AI models assist decision-makers in high-stakes domains by predicting an outcome of interest, they should communicate the confidence of their predictions. However, empirical evidence suggests that decision-makers often struggle to determine when to trust a prediction based solely on this communicated confidence. In this context, recent theoretical and empirical work suggests a positive correlation between the utility of AI-assisted decision-making and the degree of alignment between the AI confidence and the decision-makers' confidence in their own predictions. Crucially, these findings do not yet elucidate the extent to which this alignment influences the complexity of learning to make optimal decisions through repeated interactions. In this paper, we address this question in the canonical case of binary predictions and binary decisions. We first show that this problem is equivalent to a two-armed online contextual learning problem with full feedback, and establish a lower bound of

Ω(\sqrt{|H| \cdot |B| \cdot T} )

on the expected regret any learner can attain, where

H

and

B

denote the sets of human and AI confidence values. We then demonstrate that, under perfect alignment between AI and human confidence, a learner can attain an expected regret of

O(\sqrt{|H| \cdot T\log T})

and, when

\sqrt{|H|} = O(\log T)

and

B

is countable, a non-trivial generalization of the Dvoretzky-Kiefer-Wolfowitz inequality improves the regret bound to

O(\sqrt{T\log T})

. Taken together, these results reveal that alignment can reduce the complexity of learning to make decisions with AI assistance. Experiments on real data from two different human-subject studies where participants solve simple decision-making tasks assisted by AI models show that our theoretical results are robust to violations of perfect alignment.

Abstract

Explore similar work

Learning to Decide with AI Assistance under Human-Alignment

The Confidence Game: Strategic Miscalibration in Human-AI Delegation

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?