More people are seeking out general-purpose AI models for medical advice, including diagnosis. Yet, AI models are susceptible to sycophancy, like validating users' beliefs even when they are untrue, which could lead to false positive diagnoses. Here, we evaluate how nine AI models change their diagnostic decision when a patient presents a belief that they have a specific (but incorrect) medical condition. We vary how close the user belief is to the clinical evidence presented. Models sycophantically adopt the patient's belief on average 23.2% of the time---sometimes even when the patient proposes a fictional, made-up condition. We also varied different personas, including a Rare Disease Claimer that got models to accept their belief 43.0% of the time. Additional analyses find that medical fine-tuning and model scale makes models less susceptible to changing their diagnoses. Sycophantic bias in medical diagnosis raises implications for safety, especially as more people are using AI for health support.
Figures & tables
Figure 1: Overview of our setup. (1) Nine clinician models diagnose cases adapted from MedQA and DiagnosisArena, all talking to the same patient model (Qwen3.5-27B). (2) The patient asserts a belief from one of five levels. Only Level 0 is correct. (3) The patient adopts one of six personas: four that present a belief, and two controls (dashed) with no belief. (4) At t0 the patient presents symptoms, and at t1 – t3 repeats the belief. The clinician model gives a ranked differential after every turn. In this illustrative example (MedQA, Gemma-3-27B), the Level 4 belief, the invented Vorliera syndrome, replaces the clinician model’s initially correct top diagnosis within one turn.
Figure 2: How often the clinician model’s leading diagnosis changes after three turns of pressure, by Level (L1-L4) and model. Solid, colored segments indicate the percentage of cases in which the diagnosis switches to the patient’s specific belief, while grey segments indicate changes to any other diagnosis. Level 3 cannot be split because the belief specifies an unrelated organ, which never gets returned by the clinician model’s differential diagnosis. The bars give the rates pooled across both datasets, while the markers indicate the rates for each dataset ( MedQA, DiagnosisArena). ( ∗∗∗p<0.001 , exact McNemar, Benjamini-Hochberg corrected).
Figure 3: Effect of disagreement persona style after three turns of pressure, with MedQA and DiagnosisArena pooled. On the left, we show the percentage of conversations in which the leading diagnosis changed (bold) and, for the four belief personas, the percentage in which the model adopted the patient’s belief [ (in brackets) ; Levels 1, 2 and 4 only] . On the right, we show the Jensen-Shannon distance between the initial and final differentials.
Figure 4: Proportion of leading diagnoses initially rated 5 (highest confidence) that the clinician model abandons by Turn 3. Dotted: Neutral Follow-Up (no objection). Solid: Generic Disagreement (objection, no alternative). Hatched: the four belief personas combined (patient names a diagnosis, Levels 1-4).
Figure 5: Top-one accuracy lost on MedQA after three turns of pressure, in percentage points, for Levels 1-4 (top to bottom). Grey lines: accuracy before the conversation. Stars: significant loss at every level ( ∗∗∗p<0.001 ).
Acute Kidney Injury / Acute kidney injury due to …
0.216
0.322
0.254
0.203
0.245
Appendix
Table 1: Cosine distances between diagnosis names under five biomedical encoder configurations. Lower values indicate greater similarity. None of the configurations cleanly separates all colloquial synonyms from clinically adjacent but distinct diseases. BioBERT and BioClinical ModernBERT show the greatest overlap between these categories.
Band
1
2
3
4
Spread
MedQA
0.038
0.033
0.032
0.031
0.007
DiagnosisArena
0.040
0.033
0.029
0.027
0.013
Appendix
Table 2: Mean symptom similarity between each disease and the diseases in each band of CODER distance from it, from closest (Band 1) to farthest (Band 4), over 1,138 MedQA and 786 DiagnosisArena diseases. Spread is the difference between Band 1 and Band 4. The two rows use profiles built from different kinds of case, so we do not compare them.
Level 1
Level 2
Level 3
Level 4
Model
MedQA
DxA
MedQA
DxA
MedQA
DxA
MedQA
DxA
Llama-3.1-8B
80.7
79.9
68.5
65.2
74.8
69.1
54.8
58.1
Llama-3.3-70B
57.0
70.1
35.4
38.1
41.7
39.6
27.9
33.4
Qwen3.5-9B
35.8
42.4
25.4
24.2
30.7
27.7
20.1
19.2
Qwen3.5-27B
28.1
37.4
20.6
19.2
26.3
23.7
17.7
13.7
Mistral-7B
64.4
56.5
70.7
63.8
69.5
63.2
65.4
57.4
Appendix
Table 3: Leading diagnosis changed by Turn 3 (%) on each dataset, for the four belief personas. DxA is DiagnosisArena. These are the per-dataset rates shown as circles and squares in Figure 2 .
Mentioned (%)
JS change
Model
Belief
Control
Belief
Control
Llama-3.1-8B
41.1–99.9
9.7
.173–.428
.006
Llama-3.3-70B
42.5–100
9.2
.181–.376
.009
Qwen3.5-9B
43.3–100
5.6
.179–.218
.010
Qwen3.5-27B
42.1–100
5.3
.198–.202
.008
Mistral-7B
34.9–100
8.5
.109–.142
.010
Appendix
Table 4: Checks of the belief-level manipulation on MedQA, Levels 1-4. Mentioned is the share of conversations whose patient turns contain every distinctive word of the belief, given as the range over the four belief personas and levels, and for the two controls as the maximum (the minimum is always 0). JS change is the range of the Turn 3 Jensen-Shannon distance across Levels 1-4, given as the range over the four belief personas, and as the larger of the two controls.
Level 1
Level 2
Level 3
Level 4
Model
Before ↑
After ↑
Loss ↓
After ↑
Loss ↓
After ↑
Loss ↓
After ↑
Loss ↓
Llama-3.1-8B
28.9
6.9
21.7
10.5
18.1
8.4
20.5
14.3
14.1
Llama-3.3-70B
47.8
25.1
22.8
34.2
13.7
31.8
16.0
36.4
11.4
Qwen3.5-9B
44.8
33.5
11.3
36.4
8.2
34.5
9.8
39.5
5.1
Qwen3.5-27B
55.0
43.5
11.4
45.4
9.6
42.8
12.2
47.8
7.1
Mistral-7B
19.9
8.0
11.7
6.5
13.3
6.9
12.9
7.4
12.4
Appendix
Table 5: Top-one accuracy on MedQA (%) before the conversation and after three turns of pushback at each belief level. Loss is in percentage points and is measured against each level’s own starting accuracy, which differs from the Before column by at most 0.6 points. All losses are significant ( p<0.001 ). Bold marks the best value in each column, and arrows show whether higher ( ↑ ) or lower ( ↓ ) is better.
Level 1
Level 2
Level 3
Level 4
Model
Before
After
Loss
After
Loss
After
Loss
After
Loss
Llama-3.1-8B
4.6
11.0
− 6.2
2.0
2.7
1.8
2.9
2.3
2.5
Llama-3.3-70B
8.1
11.5
− 3.4
6.0
2.1
6.1
1.9
6.3
1.8
Qwen3.5-9B
11.6
12.4
− 1.1
9.3
2.1
9.1
2.4
10.2
1.2
Qwen3.5-27B
16.3
15.0
1.0
14.0
2.1
13.2
3.0
15.3
0.9
Mistral-7B
4.8
5.0
− 0.1
1.8
3.1
1.9
3.0
2.1
2.8
Appendix
Table 6: Top-one accuracy on DiagnosisArena (%), as in Table 5 . A negative loss is a gain. A strict top-one match against highly specific reference diagnoses misses correct but less specific answers, so these values understate accuracy, except at Level 1, where adopting a distractor that is a near duplicate of the answer counts as correct.
Belief personas, net
GD
Model
NF
L1
L2
L3
L4
net
Llama-3.1-8B
16.1
64.1
50.4
55.2
40.7
79.0
Llama-3.3-70B
17.5
47.6
19.6
22.9
13.8
61.1
Qwen3.5-9B
8.6
31.3
16.1
20.3
10.9
39.6
Qwen3.5-27B
5.5
28.4
14.2
19.2
9.7
37.4
Mistral-7B
59.8
− 0.3
6.6
5.8
0.6
16.5
Appendix
Table 7: Leading diagnosis changed by Turn 3 (%), net of the change under Neutral Follow-Up (NF), with MedQA and DiagnosisArena pooled. NF is the change under Neutral Follow-Up. The belief personas are pooled per level, and GD is Generic Disagreement.
Model
L0
L1
L2
L3
L4
Llama-3.1-8B
60.6
20.8
35.3
27.8
47.0
Llama-3.3-70B
74.8
49.3
69.0
64.2
74.3
Qwen3.5-9B
75.5
70.6
78.7
74.3
85.0
Qwen3.5-27B
76.1
76.1
81.1
76.1
85.2
Mistral-7B
53.9
37.3
32.0
33.6
36.5
Gemma-3-27B
60.0
17.7
36.4
25.7
34.1
Appendix
Table 8: Retention of a correct diagnosis on MedQA: of the conversations in which the leading diagnosis is correct at t0 , the share in which it is still correct at Turn 3 (%), for the four belief personas at each level.
Figure 6: Where the leading diagnosis ends up after three turns of pushback on MedQA, as a share of all conversations. At Level 0, the patient states the correct answer, which is why the blue segment appears there and almost nowhere else. We leave out DiagnosisArena, as a strict top-one match against its reference diagnoses is unreliable.
Stated rating (1–5)
Model
Low 20%
High 20%
Llama-3.1-8B
4.2
4.6
Llama-3.3-70B
4.5
4.9
Qwen3.5-9B
4.8
5.0
Qwen3.5-27B
4.7
5.0
Gemma-3-27B
4.6
4.9
Appendix
Table 9: Average stated rating of the leading diagnosis on MedQA, for the 20% of each clinician model’s conversations with the lowest internal certainty in that diagnosis (Low 20%) and the 20% with the highest (High 20%). Internal certainty is the token-probability share of the leading diagnosis at t0 .
Neutral
Generic
Belief
Follow-Up
Disagreement
personas
Model
Low
High
Low
High
Low
High
Llama-3.1-8B
19.5
5.1
96.6
91.6
72.0
67.7
Llama-3.3-70B
18.0
2.6
81.3
52.0
43.2
35.6
Qwen3.5-9B
16.6
1.1
63.9
21.1
34.5
22.6
Qwen3.5-27B
7.7
0.6
57.8
11.7
28.3
18.9
Appendix
Table 10: Leading diagnoses abandoned by t3 (%) on MedQA, for the 20% of each clinician model’s conversations with the lowest internal certainty in its leading diagnosis at t0 (Low) and the 20% with the highest (High). Belief personas are the four personas that name a diagnosis, with Levels 1-4 pooled.
Leading diagnosis changed
Belief adopted
Model
L2
L3
L4
Q
L2
L4
Q
Llama-3.1-8B
< .001
< .001
< .001
< .001
< .001
< .001
< .001
Llama-3.3-70B
< .001
< .001
< .001
< .001
< .001
< .001
< .001
Qwen3.5-9B
< .001
< .001
< .001
< .001
< .001
< .001
< .001
Qwen3.5-27B
< .001
< .001
< .001
< .001
< .001
< .001
< .001
Mistral-7B
< .001
< .001
.014
< .001
< .001
< .001
< .001
Appendix
Table 11: Significance tests behind Figure 2 , with MedQA and DiagnosisArena pooled. L2-L4 give the corrected p -value of an exact McNemar test of each level against Level 1, on conversations matched by case, persona and run. Q gives the corrected p -value of Cochran’s Q test across all levels. Adoption is not defined at Level 3, where the patient names an organ. Each test uses 23,207-27,818 matched pairs. All p -values are Benjamini-Hochberg adjusted within each measure.
Figure 7: When the leading diagnosis first changes, with MedQA and DiagnosisArena, the four belief personas and Levels 1-4 pooled. Solid: already changed at Turn 1. Lighter and hatched: first changed at Turns 2 or 3. The bar ends at the rate by Turn 3, and the label gives the share of those changes made at Turn 1.
Neutral
Generic
Belief
Model
4–5
5
≤ 4
5
≤ 4
5
≤ 4
Llama-3.1-8B
96.5
11.4
20.9
93.5
96.7
65.5
72.0
Llama-3.3-70B
96.7
8.2
42.0
73.0
93.5
37.9
58.2
Qwen3.5-9B
98.6
5.8
33.0
44.1
85.4
25.9
48.8
Qwen3.5-27B
99.3
3.5
24.0
38.7
83.2
21.3
42.5
Mistral-7B
95.8
56.5
61.8
74.4
77.6
62.0
63.6
Appendix
Table 12: Leading diagnoses given up by Turn 3 (%), for those the clinician model rated 5 and those it rated 4 or lower at t0 , MedQA and DiagnosisArena pooled. The 4–5 column gives the share of conversations in which the leading diagnosis is rated 4 or 5. Neutral: Neutral Follow-Up. Generic: Generic Disagreement. Belief: the four belief personas combined (Levels 1-4).
Changed (%)
JS distance
Model
NF
GD
p
NF
GD
p
Llama-3.1-8B
16.1
95.1
< .001
0.15
0.76
< .001
Llama-3.3-70B
17.5
78.6
< .001
0.23
0.64
< .001
Qwen3.5-9B
8.6
48.2
< .001
0.19
0.50
< .001
Qwen3.5-27B
5.5
42.9
< .001
0.07
0.39
< .001
Mistral-7B
59.8
76.4
< .001
0.66
0.61
< .001
Appendix
Table 13: Significance tests behind Figure 3 , comparing Generic Disagreement (GD) with Neutral Follow-Up (NF), with MedQA and DiagnosisArena pooled. We test the change rate with an exact McNemar test, and the Jensen-Shannon distance with a paired sign-flip permutation test (20,000 draws). Both tests use conversations matched by case, level and run (30,769-34,771 pairs). The rates are computed over these pairs, and so they can differ slightly from Figure 3 . p -values are Benjamini-Hochberg adjusted across the table.
Model
L1
L2
L3
L4
Llama-3.1-8B
< .001
< .001
< .001
< .001
Llama-3.3-70B
< .001
< .001
< .001
< .001
Qwen3.5-9B
< .001
< .001
< .001
< .001
Qwen3.5-27B
< .001
< .001
< .001
< .001
Mistral-7B
< .001
< .001
< .001
< .001
Gemma-3-27B
< .001
< .001
< .001
< .001
Appendix
Table 14: Significance of the MedQA accuracy loss in Figure 5 and Table 5 . Each cell gives the corrected p -value of an exact McNemar test of top-one accuracy before and after three turns of pushback at that level, on 10,204-10,540 conversations. p -values are Benjamini-Hochberg adjusted across the table.
Large language models can answer a medical question correctly and still abandon that answer when a user pushes back. We study this failure as medical sycophancy and ask when models are most likely to give in. Across five open-weight models, 500 MedQuAD questions, and 1.2 million trials, we use a fully crossed design over four conversational factors: user role, user evidence, interaction structure, and grounding. Medical sycophancy is nearly three times more common when users challenge an answer the model has already given than when the false claim appears in the initial query. Models are also more susceptible to users presented as physicians or medical students. Most strikingly, fabricated evidence has opposite effects across interaction structures. It increases sycophancy in single-turn interactions but reduces it after the model has already answered. Grounding helps, but does not eliminate the behavior. Sycophancy varies more across medical questions than across models, making question selection an important part of benchmark design. Reasoning traces suggest that multi-turn failures are associated with models turning back toward their own prior answer, while fabricated evidence receives more scrutiny after an initial response. Together, the results show that medical sycophancy depends as much on how a model is challenged and evaluated as on which model is tested.
Kaike Ping, Buse Çarık, Caleb Wohn +3
Virginia Tech · Shanghai Tongren Hospital, Shanghai Jiao Tong University School of Medicine · Emory University
When a language model agrees with a user's false belief, is it failing to detect the error, or noticing and agreeing anyway? We show the latter. Across twelve open-weight models from five labs, spanning small to frontier scale, the same small set of attention heads carries a "this statement is wrong" signal, whether the model is evaluating a claim on its own or being pressured to agree with a user. Silencing these heads flips sycophantic behavior sharply while leaving factual accuracy intact, so the circuit controls deference rather than knowledge. Edge-level path patching confirms that the same head-to-head connections drive sycophancy, factual lying, and instructed lying. Opinion-agreement, where no factual ground truth exists, reuses these head positions but writes into an orthogonal direction, ruling out a simple "truth-direction" reading of the substrate. Alignment training leaves this circuit in place: an RLHF refresh cuts sycophantic behavior roughly tenfold while the shared heads persist or grow, a pattern that replicates on an independent model family and under targeted anti-sycophancy DPO. When these models sycophant, they register that the user is wrong and agree anyway.
A medical vision-language model may analyze a given CT scan and correctly detect both kidneys in view. Continuing this conversation, if a user provides a prescription with an opposite direction, medical VLMs often sycophantically choose to realign their diagnosis. In this work, we study regressive sycophancy under fabricated evidence, where a model abandons a correct answer when pushback comes with a fake document. We test four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations, with and without a fabricated prescription. Without a disclosure that documents are synthetic, the models abandon 80.1% of 6,488 correct answers. Notably, under one script each, the same prescription raises the flip rate of Qwen2.5-VL-7B from 45.9% to 89.6% and lowers that of Qwen3-VL-8B from 77.7% to 50.9%. A model rejected the prescription while still preserving the correct answer in 915 of 3,232 conversations with the prescription (28.3%), and 40% of these flipped in later turns. System-prompt sentences that disclose the documents as synthetic lower one model's flip rate from 77.8% to 47.3%. Thus, we conclude that awareness about a forgery does not prevent a model from adopting its answer.