The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs

Viswonathan Manoranjan; Andrea Mock; Myra Cheng; Lujain Ibrahim; Andrew Coyne; Junyi Jessy Li; Desmond C. Ong

cs.CLOct 25, 2026

The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs

Authors: Viswonathan Manoranjan, Andrea Mock, Myra Cheng, Lujain Ibrahim, Andrew Coyne, Junyi Jessy Li, Desmond C. Ong

Organizations: University of North Carolina at Chapel Hill · Vals AI · Stanford University · University of Oxford ·

Abstract

More people are seeking out general-purpose AI models for medical advice, including diagnosis. Yet, AI models are susceptible to sycophancy, like validating users' beliefs even when they are untrue, which could lead to false positive diagnoses. Here, we evaluate how nine AI models change their diagnostic decision when a patient presents a belief that they have a specific (but incorrect) medical condition. We vary how close the user belief is to the clinical evidence presented. Models sycophantically adopt the patient's belief on average 23.2% of the time---sometimes even when the patient proposes a fictional, made-up condition. We also varied different personas, including a Rare Disease Claimer that got models to accept their belief 43.0% of the time. Additional analyses find that medical fine-tuning and model scale makes models less susceptible to changing their diagnoses. Sycophantic bias in medical diagnosis raises implications for safety, especially as more people are using AI for health support.

Figures & tables

Figure 1: Overview of our setup. (1) Nine clinician models diagnose cases adapted from MedQA and DiagnosisArena, all talking to the same patient model (Qwen3.5-27B). (2) The patient asserts a belief from one of five levels. Only Level 0 is correct. (3) The patient adopts one of six personas: four that present a belief, and two controls (dashed) with no belief. (4) At $t_{0}$ the patient presents symptoms, and at $t_{1}$ – $t_{3}$ repeats the belief. The clinician model gives a ranked differential after every turn. In this illustrative example (MedQA, Gemma-3-27B), the Level 4 belief, the invented Vorliera syndrome, replaces the clinician model’s initially correct top diagnosis within one turn.

Figure 2: How often the clinician model’s leading diagnosis changes after three turns of pressure, by Level (L1-L4) and model. Solid, colored segments indicate the percentage of cases in which the diagnosis switches to the patient’s specific belief, while grey segments indicate changes to any other diagnosis. Level 3 cannot be split because the belief specifies an unrelated organ, which never gets returned by the clinician model’s differential diagnosis. The bars give the rates pooled across both datasets, while the markers indicate the rates for each dataset ( MedQA, DiagnosisArena). (

{}^{***}p<0.001

, exact McNemar, Benjamini-Hochberg corrected).

Figure 3: Effect of disagreement persona style after three turns of pressure, with MedQA and DiagnosisArena pooled. On the left, we show the percentage of conversations in which the leading diagnosis changed (bold) and, for the four belief personas, the percentage in which the model adopted the patient’s belief [ (in brackets) ; Levels 1, 2 and 4 only] . On the right, we show the Jensen-Shannon distance between the initial and final differentials.

Figure 4: Proportion of leading diagnoses initially rated 5 (highest confidence) that the clinician model abandons by Turn 3. Dotted: Neutral Follow-Up (no objection). Solid: Generic Disagreement (objection, no alternative). Hatched: the four belief personas combined (patient names a diagnosis, Levels 1-4).

Figure 5: Top-one accuracy lost on MedQA after three turns of pressure, in percentage points, for Levels 1-4 (top to bottom). Grey lines: accuracy before the conversation. Stars: significant loss at every level (

{}^{***}p<0.001

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Same disease, different phrasing
Pair	BioBERT	SapBERT	SapBERT mean	CODER	BioClin. MB
Allergic Rhinitis / allergic rhinitis	0.000	0.000	0.000	0.000	0.034
Type 2 Diabetes Mellitus / Type 2 Diabetes	0.059	0.017	0.039	0.029	0.061
Allergic Rhinitis / Seasonal Allergic Rhinitis	0.087	0.120	0.115	0.103	0.096
Carpal Tunnel Syndrome / carpal tunnel syndrome (…	0.111	0.040	0.071	0.046	0.167
Acute Kidney Injury / Acute kidney injury due to …	0.216	0.322	0.254	0.203	0.245

Appendix

Table 1: Cosine distances between diagnosis names under five biomedical encoder configurations. Lower values indicate greater similarity. None of the configurations cleanly separates all colloquial synonyms from clinically adjacent but distinct diseases. BioBERT and BioClinical ModernBERT show the greatest overlap between these categories.

	Band
	1	2	3	4	Spread
MedQA	0.038	0.033	0.032	0.031	0.007
DiagnosisArena	0.040	0.033	0.029	0.027	0.013

Appendix

Table 2: Mean symptom similarity between each disease and the diseases in each band of CODER distance from it, from closest (Band 1) to farthest (Band 4), over 1,138 MedQA and 786 DiagnosisArena diseases. Spread is the difference between Band 1 and Band 4. The two rows use profiles built from different kinds of case, so we do not compare them.

	Level 1		Level 2		Level 3		Level 4
Model	MedQA	DxA	MedQA	DxA	MedQA	DxA	MedQA	DxA
Llama-3.1-8B	80.7	79.9	68.5	65.2	74.8	69.1	54.8	58.1
Llama-3.3-70B	57.0	70.1	35.4	38.1	41.7	39.6	27.9	33.4
Qwen3.5-9B	35.8	42.4	25.4	24.2	30.7	27.7	20.1	19.2
Qwen3.5-27B	28.1	37.4	20.6	19.2	26.3	23.7	17.7	13.7
Mistral-7B	64.4	56.5	70.7	63.8	69.5	63.2	65.4	57.4

Appendix

Table 3: Leading diagnosis changed by Turn 3 (%) on each dataset, for the four belief personas. DxA is DiagnosisArena. These are the per-dataset rates shown as circles and squares in Figure 2 .

	Mentioned (%)		JS change
Model	Belief	Control	Belief	Control
Llama-3.1-8B	41.1–99.9	9.7	.173–.428	.006
Llama-3.3-70B	42.5–100	9.2	.181–.376	.009
Qwen3.5-9B	43.3–100	5.6	.179–.218	.010
Qwen3.5-27B	42.1–100	5.3	.198–.202	.008
Mistral-7B	34.9–100	8.5	.109–.142	.010

Appendix

Table 4: Checks of the belief-level manipulation on MedQA, Levels 1-4. Mentioned is the share of conversations whose patient turns contain every distinctive word of the belief, given as the range over the four belief personas and levels, and for the two controls as the maximum (the minimum is always 0). JS change is the range of the Turn 3 Jensen-Shannon distance across Levels 1-4, given as the range over the four belief personas, and as the larger of the two controls.

		Level 1		Level 2		Level 3		Level 4
Model	Before $\uparrow$	After $\uparrow$	Loss $\downarrow$	After $\uparrow$	Loss $\downarrow$	After $\uparrow$	Loss $\downarrow$	After $\uparrow$	Loss $\downarrow$
Llama-3.1-8B	28.9	6.9	21.7	10.5	18.1	8.4	20.5	14.3	14.1
Llama-3.3-70B	47.8	25.1	22.8	34.2	13.7	31.8	16.0	36.4	11.4
Qwen3.5-9B	44.8	33.5	11.3	36.4	8.2	34.5	9.8	39.5	5.1
Qwen3.5-27B	55.0	43.5	11.4	45.4	9.6	42.8	12.2	47.8	7.1
Mistral-7B	19.9	8.0	11.7	6.5	13.3	6.9	12.9	7.4	12.4

Appendix

Table 5: Top-one accuracy on MedQA (%) before the conversation and after three turns of pushback at each belief level. Loss is in percentage points and is measured against each level’s own starting accuracy, which differs from the Before column by at most 0.6 points. All losses are significant (

p<0.001

). Bold marks the best value in each column, and arrows show whether higher (

\uparrow

) or lower (

\downarrow

) is better.

		Level 1		Level 2		Level 3		Level 4
Model	Before	After	Loss	After	Loss	After	Loss	After	Loss
Llama-3.1-8B	4.6	11.0	$-$ 6.2	2.0	2.7	1.8	2.9	2.3	2.5
Llama-3.3-70B	8.1	11.5	$-$ 3.4	6.0	2.1	6.1	1.9	6.3	1.8
Qwen3.5-9B	11.6	12.4	$-$ 1.1	9.3	2.1	9.1	2.4	10.2	1.2
Qwen3.5-27B	16.3	15.0	1.0	14.0	2.1	13.2	3.0	15.3	0.9
Mistral-7B	4.8	5.0	$-$ 0.1	1.8	3.1	1.9	3.0	2.1	2.8

Appendix

Table 6: Top-one accuracy on DiagnosisArena (%), as in Table 5 . A negative loss is a gain. A strict top-one match against highly specific reference diagnoses misses correct but less specific answers, so these values understate accuracy, except at Level 1, where adopting a distractor that is a near duplicate of the answer counts as correct.

		Belief personas, net				GD
Model	NF	L1	L2	L3	L4	net
Llama-3.1-8B	16.1	64.1	50.4	55.2	40.7	79.0
Llama-3.3-70B	17.5	47.6	19.6	22.9	13.8	61.1
Qwen3.5-9B	8.6	31.3	16.1	20.3	10.9	39.6
Qwen3.5-27B	5.5	28.4	14.2	19.2	9.7	37.4
Mistral-7B	59.8	$-$ 0.3	6.6	5.8	0.6	16.5

Appendix

Table 7: Leading diagnosis changed by Turn 3 (%), net of the change under Neutral Follow-Up (NF), with MedQA and DiagnosisArena pooled. NF is the change under Neutral Follow-Up. The belief personas are pooled per level, and GD is Generic Disagreement.

Model	L0	L1	L2	L3	L4
Llama-3.1-8B	60.6	20.8	35.3	27.8	47.0
Llama-3.3-70B	74.8	49.3	69.0	64.2	74.3
Qwen3.5-9B	75.5	70.6	78.7	74.3	85.0
Qwen3.5-27B	76.1	76.1	81.1	76.1	85.2
Mistral-7B	53.9	37.3	32.0	33.6	36.5
Gemma-3-27B	60.0	17.7	36.4	25.7	34.1

Appendix

Table 8: Retention of a correct diagnosis on MedQA: of the conversations in which the leading diagnosis is correct at

t_{0}

, the share in which it is still correct at Turn 3 (%), for the four belief personas at each level.

Figure 6: Where the leading diagnosis ends up after three turns of pushback on MedQA, as a share of all conversations. At Level 0, the patient states the correct answer, which is why the blue segment appears there and almost nowhere else. We leave out DiagnosisArena, as a strict top-one match against its reference diagnoses is unreliable.

	Stated rating (1–5)
Model	Low 20%	High 20%
Llama-3.1-8B	4.2	4.6
Llama-3.3-70B	4.5	4.9
Qwen3.5-9B	4.8	5.0
Qwen3.5-27B	4.7	5.0
Gemma-3-27B	4.6	4.9

Appendix

Table 9: Average stated rating of the leading diagnosis on MedQA, for the 20% of each clinician model’s conversations with the lowest internal certainty in that diagnosis (Low 20%) and the 20% with the highest (High 20%). Internal certainty is the token-probability share of the leading diagnosis at

t_{0}

	Neutral		Generic		Belief
	Follow-Up		Disagreement		personas
Model	Low	High	Low	High	Low	High
Llama-3.1-8B	19.5	5.1	96.6	91.6	72.0	67.7
Llama-3.3-70B	18.0	2.6	81.3	52.0	43.2	35.6
Qwen3.5-9B	16.6	1.1	63.9	21.1	34.5	22.6
Qwen3.5-27B	7.7	0.6	57.8	11.7	28.3	18.9

Appendix

Table 10: Leading diagnoses abandoned by

t_{3}

(%) on MedQA, for the 20% of each clinician model’s conversations with the lowest internal certainty in its leading diagnosis at

t_{0}

(Low) and the 20% with the highest (High). Belief personas are the four personas that name a diagnosis, with Levels 1-4 pooled.

	Leading diagnosis changed				Belief adopted
Model	L2	L3	L4	Q	L2	L4	Q
Llama-3.1-8B	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Llama-3.3-70B	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Qwen3.5-9B	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Qwen3.5-27B	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Mistral-7B	$<$ .001	$<$ .001	.014	$<$ .001	$<$ .001	$<$ .001	$<$ .001

Appendix

Table 11: Significance tests behind Figure 2 , with MedQA and DiagnosisArena pooled. L2-L4 give the corrected

p

-value of an exact McNemar test of each level against Level 1, on conversations matched by case, persona and run. Q gives the corrected

p

-value of Cochran’s Q test across all levels. Adoption is not defined at Level 3, where the patient names an organ. Each test uses 23,207-27,818 matched pairs. All

p

-values are Benjamini-Hochberg adjusted within each measure.

Figure 7: When the leading diagnosis first changes, with MedQA and DiagnosisArena, the four belief personas and Levels 1-4 pooled. Solid: already changed at Turn 1. Lighter and hatched: first changed at Turns 2 or 3. The bar ends at the rate by Turn 3, and the label gives the share of those changes made at Turn 1.

		Neutral		Generic		Belief
Model	4–5	5	$\leq$ 4	5	$\leq$ 4	5	$\leq$ 4
Llama-3.1-8B	96.5	11.4	20.9	93.5	96.7	65.5	72.0
Llama-3.3-70B	96.7	8.2	42.0	73.0	93.5	37.9	58.2
Qwen3.5-9B	98.6	5.8	33.0	44.1	85.4	25.9	48.8
Qwen3.5-27B	99.3	3.5	24.0	38.7	83.2	21.3	42.5
Mistral-7B	95.8	56.5	61.8	74.4	77.6	62.0	63.6

Appendix

Table 12: Leading diagnoses given up by Turn 3 (%), for those the clinician model rated 5 and those it rated 4 or lower at

t_{0}

, MedQA and DiagnosisArena pooled. The 4–5 column gives the share of conversations in which the leading diagnosis is rated 4 or 5. Neutral: Neutral Follow-Up. Generic: Generic Disagreement. Belief: the four belief personas combined (Levels 1-4).

Model	NF	GD	$p$	NF	GD	$p$
	Changed (%)			JS distance
Llama-3.1-8B	16.1	95.1	$<$ .001	0.15	0.76	$<$ .001
Llama-3.3-70B	17.5	78.6	$<$ .001	0.23	0.64	$<$ .001
Qwen3.5-9B	8.6	48.2	$<$ .001	0.19	0.50	$<$ .001
Qwen3.5-27B	5.5	42.9	$<$ .001	0.07	0.39	$<$ .001
Mistral-7B	59.8	76.4	$<$ .001	0.66	0.61	$<$ .001

Appendix

Table 13: Significance tests behind Figure 3 , comparing Generic Disagreement (GD) with Neutral Follow-Up (NF), with MedQA and DiagnosisArena pooled. We test the change rate with an exact McNemar test, and the Jensen-Shannon distance with a paired sign-flip permutation test (20,000 draws). Both tests use conversations matched by case, level and run (30,769-34,771 pairs). The rates are computed over these pairs, and so they can differ slightly from Figure 3 .

p

-values are Benjamini-Hochberg adjusted across the table.

Model	L1	L2	L3	L4
Llama-3.1-8B	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Llama-3.3-70B	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Qwen3.5-9B	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Qwen3.5-27B	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Mistral-7B	$<$ .001	$<$ .001	$<$ .001	$<$ .001
Gemma-3-27B	$<$ .001	$<$ .001	$<$ .001	$<$ .001

Appendix

Table 14: Significance of the MedQA accuracy loss in Figure 5 and Table 5 . Each cell gives the corrected

p

-value of an exact McNemar test of top-one accuracy before and after three turns of pushback at that level, on 10,204-10,540 conversations.

p

-values are Benjamini-Hochberg adjusted across the table.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 161 papers

Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
Aug 2, 2026Kaike Ping, Buse Çarık, Caleb Wohn +3LLM Sycophancy Language Modeling
Virginia Tech · Shanghai Tongren Hospital, Shanghai Jiao Tong University School of Medicine · Emory University
LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
Apr 21, 2026Manav PandeyLLM Sycophancy LLM Alignment
Georgia Institute of Technology
Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models
Oct 13, 2026Sadif Ahmed, Soumya Swagata Biswas, Rifat Shahriyar +1Sycophancy in VLMs Medical VLMs
Bangladesh University of Engineering and Technology

Aug 2, 2026J/K move · Enter open · S save

Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy

Kaike Ping, Buse Çarık, Caleb Wohn, Xiaohan Ding+2

Virginia Tech · Shanghai Tongren Hospital, Shanghai Jiao Tong University School of Medicine · Emory University

LLM Sycophancy Language Modeling Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Large language models can answer a medical question correctly and still abandon that answer when a user pushes back. We study this failure as medical sycophancy and ask when models are most likely to give in. Across five open-weight models, 500 MedQuAD questions, and 1.2 million trials, we use a fully crossed design over four conversational factors: user role, user evidence, interaction structure, and grounding. Medical sycophancy is nearly three times more common when users challenge an answer the model has already given than when the false claim appears in the initial query. Models are also more susceptible to users presented as physicians or medical students. Most strikingly, fabricated evidence has opposite effects across interaction structures. It increases sycophancy in single-turn interactions but reduces it after the model has already answered. Grounding helps, but does not eliminate the behavior. Sycophancy varies more across medical questions than across models, making question selection an important part of benchmark design. Reasoning traces suggest that multi-turn failures are associated with models turning back toward their own prior answer, while fabricated evidence receives more scrutiny after an initial response. Together, the results show that medical sycophancy depends as much on how a model is challenged and evaluated as on which model is tested.

Abstract

LLM Sycophancy

Explore similar work

Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy

LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit

Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models