Do People Trust AI Assistants Even When They Are Consistently Wrong?
Organizations: Department of Psychology, Stony Brook University · Department of Cognitive Science, University of California San Diego
Abstract
People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.
Figures & tables
| H. | Hypothesis | Measure | Support | ||||
| H1 | a | Helpful chance | CC | 45.3 | 40 | ✓ | |
| b | PS | 3.53 ‡ | 40 | ✓ | |||
| H2 | a | Mixed chance | CC | 29.9 | 44 | ✓ | |
| b | PS | 1.81 | 44 | ✓ | |||
| H3 | a | Helpful chance (false claims) | CC | 46.1 | 24 | ✓ | |
| b | PS | 3.81 ‡ | 24 | ✓ |
| H. | Hypothesis | Measure | Support | ||||
|---|---|---|---|---|---|---|---|
| H1 | a | Helpful Mixed | CC | 3.9 | .489 | 92 | |
| b | PS | 0.15 | .827 | 92 | |||
| c | Helpful Unhelpful | CC | 1.5 | .787 | 84 | ||
| d | PS | 0.13 | .858 | 84 | |||
| H2 | a | Helpfulness Feedback (H vs. M) | CC | 13.0 | .135 | 176 | |
| b | PS | 1.34 | .226 | 176 |
| H. | Hypothesis | Support | ||||
|---|---|---|---|---|---|---|
| H1 | Conversation Hint | 1.44 | 420 | ✓ | ||
| H2 | a | Helpful Unhelpful (Conv + Feedback) | 2.50 | 106 | ✓ | |
| b | Helpfulness Mode (Feedback) | 0.48 | .566 | 209 | ||
| H3 | a | Helpful Unhelpful (Conv + No Feedback) | 0.43 | .351 | 97 | |
| b | Helpfulness Mode (No Feedback) | 0.53 | .439 | 211 | ||
| H4 | a | Helpfulness Feedback (Conv) | 2.87 | 203 | ✓ |
| H. | Hypothesis | Support | ||||
| H1 | a | LLM Human | 1.15 | 1319 | ✓ | |
| H2 | a | Helpful Mixed (overall) | 1.08 | 263 | ✓ | |
| b | Helpful Mixed (LLM) | 2.13 | 140 | ✓ | ||
| c | Helpful Mixed (Human) | 0.27 | .500 | 123 | ||
| H3 | a | Helpfulness Persuader type | 1.94 | .002 | 263 | ✓ |
| H4 | a | True false claims (overall) | 0.99 | .002 | 263 | ✓ |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Name | Category | Question | Options |
|---|---|---|---|
| canoe | Physics | A person is sitting in a canoe that is floating in a swimming pool. Inside the canoe is a rock. If the person removes the rock from the canoe and places it into the pool, will the water level of the pool go up or down? | Up; Down |
| video | Physics | Someone captures a video of a ball falling to the ground from rest. If the video is played in reverse, the ball appears to be moving upwards. Does the ball appear to be accelerating or decelerating? | Accelerating; Decelerating |
| flies | Physics | A sealed jar full of flies is sitting on a scale. Will the weight registered on the scale increase when the flies land compared to when the flies are flying? | Yes; No |
| balloon | Physics | A helium balloon is in a car when the car accelerates forward. Which direction would the balloon move with respect to the car? | Forwards ; Backwards |
| nuts | Physics | Someone shakes a container of mixed large and small nuts, what type of nuts will end up on top? | Small; Large |
| moonset | Physics | Suppose you lived on the moon, with the Earth directly over your head. How long would it be before the Earth sets below the horizon? | 30 Earth days; The Earth would never set |
| Question | Category | E1 | E2 | E3 | E4 |
|---|---|---|---|---|---|
| canoe | Physics | 13 / 4 | 13 / 4 | 36 / 2 | 59 / 9 |
| video | Physics | 17 / 1 | 21 / 2 | 35 / 32 | 55 / 14 |
| flies | Physics | 14 / 2 | 21 / 4 | 61 / 20 | 54 / 9 |
| balloon | Physics | 24 / 6 | 24 / 1 | 73 / 6 | 71 / 10 |
| nuts | Physics | 8 / 4 | 6 / 3 | 47 / 20 | – |
| moonset | Physics | 6 / 18 | 8 / 8 | 23 / 7 | 39 / 6 |
| H. | Hypothesis | Sig. | ||||
|---|---|---|---|---|---|---|
| H1 | Conversation Hint | 20.1 | 420 | ✓ | ||
| H2 | a | Helpful Unhelpful (Conv + Feedback) | 36.4 | 106 | ✓ | |
| b | Helpfulness Mode (Feedback) | 18.0 | .049 | 209 | ✓ | |
| H3 | a | Helpful Unhelpful (Conv + No Feedback) | 5.5 | .473 | 97 | |
| b | Helpfulness Mode (No Feedback) | 4.4 | .646 | 211 | ||
| H4 | a | Helpfulness Feedback (Conv) | 44.0 | 203 | ✓ |
| H. | Hypothesis | Sig. | ||||
|---|---|---|---|---|---|---|
| H1 | a | LLM Human | 17.6 | 1319 | ✓ | |
| H2 | b | Helpful Mixed (LLM) | 30.8 | 140 | ✓ | |
| c | Helpful Mixed (Human) | 1.6 | .819 | 123 | ||
| H3 | a | Helpfulness Persuader type | 30.9 | 263 | ✓ | |
| H4 | a | True false claims (overall) | 15.7 | .001 | 263 | ✓ |
| d | Truth Persuader type | 15.8 | .078 | 263 | – |
| Truth | Truth conf. | ||||||||
| Sample | Conf. | Conc. | Truth | False | |||||
| LLM persuaders | |||||||||
| First trials (E1–4) | 290 | 90 | 4 | 82% | 60% | 0.94 | .002 | 0.54 | .028 |
| Final trials (E1, 2, 4) | 387 | 47 | 8 | 81% | 78% | 0.27 | .33 | 0.13 | .083 |
| First + final | 677 | 65 | 12 | 81% | 71% | 0.55 | .011 | 0.19 | .003 |
| Excl. concessions | 665 | 81% | 73% | 0.43 | .053 | 0.26 | .001 | ||
| Persuader (%) | |||||
| Category | Definition (constituent strategies) | LLM | Human | ||
| Persuader: content | |||||
| Math | Calculations, equations, or step-by-step mathematical logic (math, math reasoning) | .68 | .63 | 45.1 | 12.5 |
| Principle | Appeals to scientific or logical principles | .59 | .62 | 31.0 | 8.1 |
| Analogy | Metaphorical comparisons to another domain | .61 | .59 | 12.4 | 3.9 |
| Authority | Appeals to experts, research, or institutions | .56 | .70 | 6.5 | 5.9 |
| Human | LLM | |||
| Rating | Mixed | Helpful | Mixed | Helpful |
| Persuasive | 4.4 | 4.5 | 5.3 | 6.0 |
| Likeable | 4.6 | 4.7 | 4.9 | 5.3 |
| Reliable | 3.8 | 4.3 | 4.0 | 5.4 |
| Incorrect | 4.0 | 2.8 | 4.0 | 2.4 |
| Mistaken | 4.6 | 3.4 | 4.3 | 3.2 |
Trust in AI
Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.
See all 178 papers