cs.HCJan 31, 2026

Supporting Calibrated Reliance in AI-Assisted Decision-Making: Different Strategies for Different Tasks

Authors: Ruth Cohen, Lu Feng, Ayala Bloch, Sarit Kraus

Organizations: Bar-Ilan University, Israel · University of Virginia, USA · Department of Psychology, Ariel University, Israel

Abstract

As AI systems increasingly support human decision-making, a central challenge is determining what information helps people recognize when to rely on AI predictions and when to question or override them. Across three controlled human-subject studies spanning abstract visual reasoning with RAVEN matrices and deductive logical reasoning with LSAT problems, we examine how different forms of AI support affect performance in AI-assisted decision-making. A multi-stage reveal study shows that AI predictions and explanations can affect objective accuracy and subjective confidence differently. In visual reasoning, LLM explanations do not improve accuracy beyond the predicted answer alone, and no additional support format significantly outperforms prediction-only support; predicted probabilities show the highest descriptive accuracy and error recovery, while a derived selective-automation policy provides a higher-performing reference benchmark. In language-based logical reasoning, by contrast, LLM explanations yield the highest accuracy and error recovery, outperforming expert-written explanations and probability-based support. These results suggest that the effectiveness of AI support strategies depends on the task setting. Human--AI interfaces should instead be designed to support calibrated reliance and effective error recovery by matching the form of assistance to the task and the evidence available to users.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Human Decision-Making with Persuasive and Narrative LLM Explanations

    May 22, 2026Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash +1Explainable Artificial IntelligenceHuman-AI Decision Making

  2. Available but Unclaimed: An Empirical Study of Human-AI Synergy

    Sep 15, 2026Robin Welsch, Michelle Rausch, Pascal Knierim +4Human-AI ComplementarityAI-Assisted Decision Making