cs.HCNov 10, 2026

Do People Trust AI Assistants Even When They Are Consistently Wrong?

Authors: Cameron R. Jones, Julia Wong, Aaron S. Cohen, James Woolley, Benjamin K. Bergen

Organizations: Department of Psychology, Stony Brook University · Department of Cognitive Science, University of California San Diego

Abstract

People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

    Oct 22, 2026Deepthika S, Pawni Yadav, B. SankarTrust in AILLM Reliability

  2. Large language models can effectively convince people to believe conspiracies

    Jan 8, 2026Thomas H. Costello, Kellin Pelrine, Matthew Kowal +6Deception in Language ModelsLLM Safety

  3. When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judge

    Aug 21, 2026Xin Sun, Di Wu, Yuchen Guo +4LLM-as-a-JudgeTrust in AI