As AI systems increasingly support human decision-making, a central challenge is determining what information helps people recognize when to rely on AI predictions and when to question or override them. Across three controlled human-subject studies spanning abstract visual reasoning with RAVEN matrices and deductive logical reasoning with LSAT problems, we examine how different forms of AI support affect performance in AI-assisted decision-making. A multi-stage reveal study shows that AI predictions and explanations can affect objective accuracy and subjective confidence differently. In visual reasoning, LLM explanations do not improve accuracy beyond the predicted answer alone, and no additional support format significantly outperforms prediction-only support; predicted probabilities show the highest descriptive accuracy and error recovery, while a derived selective-automation policy provides a higher-performing reference benchmark. In language-based logical reasoning, by contrast, LLM explanations yield the highest accuracy and error recovery, outperforming expert-written explanations and probability-based support. These results suggest that the effectiveness of AI support strategies depends on the task setting. Human--AI interfaces should instead be designed to support calibrated reliance and effective error recovery by matching the form of assistance to the task and the evidence available to users.
Figures & tables
Figure 1. The multi-stage human–AI decision process used to separate the effects of the AI prediction and its explanation on user decisions.
Figure 2. Overall accuracy and self-reported confidence across the three reveal stages in the multi-stage RAVEN study. Accuracy improves after the AI prediction but plateaus once explanations are introduced, while confidence increases only after explanations are presented.
Figure 3. Relationship between agreement with correct AI predictions and recovery from incorrect AI predictions across AI-support conditions in the RAVEN task. The selective-automation policy is a derived benchmark based on responses from the predicted-probability condition rather than an independent participant group.
Figure 4. Objective accuracy across the five participant-facing conditions in the RAVEN task, with the derived selective-automation policy shown as a reference benchmark.
RAVEN Study
LSAT Study
LLM
OS
Probability
LLM
Expert
Probability
Understanding
4.10
3.52
4.21
4.40
3.85
3.26
Clarity
3.50
3.43
4.42
4.40
3.70
3.74
Trust
3.95
3.67
4.54
4.45
3.80
4.00
Table 1. Mean subjective ratings (1–5 Likert scale) of perceived understanding, clarity, and trust in the RAVEN and LSAT studies. Ratings are reported only for conditions that provided additional information beyond the AI prediction.
Figure 5. Relationship between agreement with correct AI predictions and recovery from incorrect AI predictions across support conditions in the LSAT task.
Figure 6. Objective accuracy across human–AI support conditions in the LSAT task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Gender
Age
Study
Participants
Women
Men
Mean
SD
RAVEN multi-stage
27
8
19
27.26
4.89
RAVEN between-subjects
100
35
65
24.42
6.27
LSAT between-subjects
80
26
54
28.25
6.73
Appendix
Table 2. Participant demographics across the three studies.
Figure 7. Illustrative user interface examples from the LSAT study.
Artificial Intelligence (AI) tools are widely used to support decision making in tasks and domains where no immediate performance feedback is available. In these settings, users cannot learn to adjust their reliance behavior over time through trial and error. However, little is known about how novice users calibrate reliance on AI when external feedback is unavailable, or whether AI explanations can support calibration in its absence. We introduce reliance calibration as an organizing construct for studying how novice users dynamically adjust reliance behavior, and examine how AI explanations and meta-cognitive self-assessment shape it. Through a between-subjects study with 110 participants completing a clinical entity extraction task with AI assistance and limited performance feedback, we observe that novice users exhibit systematic drift toward over-reliance in the presence of explanations, while higher self-reported task understanding is associated with more selective reliance behavior. These results extend reliance calibration research into human-AI collaboration contexts without real-time performance signals and present actionable guidelines on designing AI tools that must support appropriate reliance in these settings.
Eun Jeong Kang, Peter, Duan +1
Xianpi · McMaster University Hamilton, Ontario, Canada
Large language models (LLMs) have the potential to aid and improve human decision-making in classification tasks, not only by providing fairly accurate predictions, but also in their ability to generate cogent narrative explanations of those predictions. Prior work has demonstrated that people generally find AI narrative explanations to be understandable, trustworthy, and convincing for changing beliefs and opinions; however, less is known about the impact of narrative explanations on objective human decision-making performance. Here we conduct a large-scale human behavioral experiment to evaluate decision-making performance with LLM-generated narrative explanations of varying persuasiveness. We found the degree of persuasiveness, or lack thereof, for LLM-based explanations did not meaningfully impact decision accuracy over a simple AI prediction alone, in agreement with typical results with explainable AI based on feature importance. We found evidence that narratives increased reliance on AI, but both when the AI prediction was correct and incorrect. Exploratory analyses also indicated that the more persuasive narratives may have had a detrimental effect on decision response times and the ability to discriminate between a correct and incorrect AI prediction. Overall, this work indicates that including narrative explanations with AI predictions may involve tradeoffs for decision-making performance, and more work is needed to determine how and when narrative explanations impact human decision-making.
Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash +1
DEVCOM Army Research Laboratory Adelphi, MD 20783 · University of Texas at Dallas Richardson, TX 75080 · Virginia Polytechnic Institute and State University Blacksburg, VA 24060
People increasingly reason with large language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted-unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent reasoning.
Robin Welsch, Michelle Rausch, Pascal Knierim +4
Aalto University, Espoo, Finland · University of Innsbruck, Innsbruck, Austria · Humboldt-Universität zu Berlin, Berlin, Germany +1