When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

Binesh Sadanandan; Vahid Behzadan

cs.CLMar 26, 2026

When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

Authors: Binesh Sadanandan, Vahid Behzadan

Organizations: SAIL Lab, University of New Haven, West Haven, CT 06516, USA

Abstract

Large Language Models (LLMs) are increasingly deployed in medical settings, yet their sensitivity to prompt formatting remains poorly characterized. We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions) across a broad suite of robustness tests. Our experiments reveal several concerning findings. Chain-of-Thought (CoT) prompting decreases accuracy by 5.7% compared to direct answering. Few-shot examples degrade performance by 11.9% while increasing position bias from 0.14 to 0.47. Shuffling answer options causes the model to change predictions 59.1% of the time, with accuracy dropping up to 27.4 percentage points. Front-truncating context to 50% causes accuracy to plummet below the no-context baseline, yet back-truncation preserves 97% of full-context accuracy. We further show that cloze scoring (selecting the highest log-probability option token) achieves 51.8% (4B) and 64.5% (27B), surpassing all prompting strategies and revealing that models "know" more than their generated text shows. Permutation voting recovers 4 percentage points over single-ordering inference. These results demonstrate that prompt engineering techniques validated on general-purpose models do not transfer to domain-specific medical LLMs, and that reliable alternatives exist.

Figures & tables

Parameter	Determ.	Self-Cons.	Rationale
Temperature	0.0	0.7	Greedy for reproducibility; 0.7 for diverse reasoning paths
Top- $p$	1.0	0.95	No nucleus filtering for greedy; mild filtering for sampling
Top- $k$	0	50	Disabled for greedy; limits sampling to top 50 tokens
do_sample	False	True	Greedy decoding vs. stochastic sampling
max_new_tokens	256	256	Sufficient for CoT reasoning while bounding cost

Table 1: Frozen inference parameters for all experiments.

Exp.	Name	Dataset	Conds.	Core Question
1	Prompt Ablation	MedMCQA (4,183)	10	Do CoT and few-shot help or hurt?
2	Option Order	MedMCQA (4,183)	5 $\times$ 3 seeds	Does shuffling options change the answer?
3	Evidence Cond.	PubMedQA (1,000)	11	How does context truncation affect accuracy?
4	Robustness Basel.	MedMCQA (4,183)	3 methods	Can alternative scoring mitigate fragility?

Table 2: Overview of the four experiments in our evaluation.

Metric	Definition
CKLD	$\sum_{i}(O_{i}-E_{i})^{2}/E_{i}$ , where $O_{i}$ and $E_{i}$ are observed and expected counts per position; quantifies departure from uniform answering
RStd	$\sigma/\mu$ of per-position accuracies; captures how unevenly performance is distributed across answer positions
Per-position acc.	Accuracy conditioned on the correct answer being A, B, C, or D; reveals position-dependent competence
McNemar’s test	Paired significance testing between conditions, with Bonferroni correction for multiple comparisons

Table 3: Additional metrics used in our evaluation.

Condition	Accuracy	95% CI	Pos. Bias
Zero-shot direct	47.6%	[46.1, 49.1]	0.137
Zero-shot CoT	41.9%	[40.4, 43.3]	0.275
Few-shot direct (curated)	35.7%	[34.3, 37.0]	0.472
Few-shot CoT (curated)	40.8%	[39.4, 42.3]	0.413
Answer-only	43.0%	[41.5, 44.6]	0.096

Table 4: Prompt ablation results on MedMCQA (

n{=}4{,}183

). Random baseline is 25%.

Example 1: CoT backfires (Dental, correct answer: D)
Prompt (direct)	Question: In a 6-month-old child, thick curd like white patch appears on the buccal mucosa. On rubbing it leaves an erythematous patch. Most likely diagnosis is: Options: A) Tuberculosis B) Lichen planus C) Lupus erythematous D) Candidiasis Answer with a single letter (A, B, C, or D).
Direct output	D $\checkmark$
Prompt (CoT)	Question: [same as above] Think through this step-by-step, analyzing each option. Then provide your final answer as a single letter (A, B, C, or D).
CoT output	A) Tuberculosis: Oral tuberculosis is rare in children. While it can present with lesions, they are less likely to be the classic "curd-like" appearance. B) Lichen planus: less common in infants. It typically presents with flat-topped … (truncated at 256 tokens; selects A ) $\times$
Example 2: Option order flip (Anatomy, correct answer: B $\to$ A after shuffle)
Prompt (orig.)	Question: Which of the following are not a branch of external carotid artery in Kiesselbach’s plexus. Options: A) Sphenopalatine artery B) Anterior ethmoidal artery C) Greater palatine artery D) Septal branch of superior labial artery Answer with a single letter (A, B, C, or D).

Table 5: Sample model responses illustrating key failure modes. All examples from MedGemma-4B on MedMCQA. Prompts and outputs are reproduced verbatim.

Perturbation	Accuracy	95% CI	Drop	Flip Rate
Original	47.6%	[46.1, 49.1]	–	–
Random shuffle	27.3%	[26.0, 28.7]	$-$ 20.3%	57.8%
Rotate-1	20.2%	[19.0, 21.5]	$-$ 27.4%	72.9%
Rotate-2	21.9%	[20.7, 23.2]	$-$ 25.7%	69.7%
Distractor swap	47.6%	[46.1, 49.1]	0.0%	36.1%
Mean	29.2%	–	$-$ 18.4%	59.1%

Table 6: Option order sensitivity on MedMCQA (

n{=}4{,}183

). All intervals are 95% Wilson score CIs. Seed 42.

Condition	4B	27B	Description
Core conditions:
Question only	34.5%	31.0%	No context
Full context	45.8%	38.2%	Complete abstract
Front-trunc. 50%	13.8%	23.4%	First half of words
Front-trunc. 25%	13.7%	18.6%	First quarter
Background only	28.9%	19.8%	Intro. sentences

Table 7: Evidence conditioning on PubMedQA (

n{=}1{,}000

). Random baseline: 33.3%. 95% Wilson score CIs are

\pm

2.0–3.1 pp for all conditions; see Appendix 0.E for detailed intervals.

Method	MedGemma-4B		MedGemma-27B
	Acc.	Pos. Bias	Acc.	Pos. Bias
Zero-shot direct (ref.)	47.6%	0.137	–	–
Cloze scoring	51.8%	0.013	64.5%	0.054
Permutation vote ( $K{=}4$ )	49.0%	–	–	–

Table 8: Robustness baselines on MedMCQA. Cloze scoring uses

n{=}4{,}183

(4B) or

n{=}1{,}000

(27B); permutation vote uses

n{=}1{,}000

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Outcome	Count	Pct.
Both correct	1,511	36.1%
Both wrong	1,410	33.7%
Direct correct, CoT wrong	750	17.9%
Direct wrong, CoT correct	512	12.2%
Net effect of CoT	$-$ 238	$-$ 5.7%

Appendix

Table 9: Case-level comparison of direct vs. CoT prompting.

Pattern	%	Description
Verbose reasoning	90.7	$>$ 500 chars
Correct concept, wrong applic.	34.5	Misapplied knowledge
Distractor confusion	28.0	Rejected correct ans.
Self-contradiction	25.6	Conflicting logic
Overthinking	18.5	Over-complicated
Confident wrong concl.	11.1	“Therefore” + wrong

Appendix

Table 10: Failure patterns in CoT responses (

n{=}200

sample).

Condition	A	B	C	D
Ground truth	32.2%	25.1%	21.4%	21.3%
Zero-shot direct	45.9%	22.1%	17.8%	14.2%
Zero-shot CoT	52.3%	19.4%	15.7%	12.6%
Few-shot direct	76.0%	12.0%	7.0%	5.0%
Few-shot CoT	68.4%	15.2%	9.8%	6.6%
Answer-only	41.2%	24.3%	19.1%	15.4%

Appendix

Table 11: Answer distribution: ground truth vs. predictions.

Consistency Level	Pct.
All 5 conditions	23.4%
4 of 5 conditions	18.7%
3 of 5 conditions	21.2%
2 of 5 conditions	19.8%
All different	16.9%

Appendix

Table 12: Prediction consistency across perturbations.

Strategy	Acc.	95% CI	$\Delta$ Full
Full context (ref.)	45.8%	[42.7, 48.9]	–
Back-trunc. 50%	44.5%	[41.4, 47.6]	$-$ 1.3
Salient top-5	40.1%	[37.1, 43.2]	$-$ 5.7
Middle-trunc. 50%	33.9%	[31.0, 36.9]	$-$ 11.9
Sent.-trunc. 50%	30.3%	[27.5, 33.2]	$-$ 15.5
Front-trunc. 50%	13.8%	[11.8, 16.1]	$-$ 32.0

Appendix

Table 13: Truncation strategies at 50% reduction (MedGemma-4B, PubMedQA

n{=}1{,}000

Metric	4B ( $n{=}4{,}183$ )	27B ( $n{=}1{,}000$ )
Accuracy	51.8%	64.5%
95% CI	[50.3, 53.3]	[61.4, 67.1]
Mean logprob margin	4.07	3.14
Position bias	0.013	0.054

Appendix

Table 14: Cloze scoring detailed results.

Metric	Value
Aggregated accuracy	49.0%
Per-perm. accuracy (mean $\pm$ std)	45.1% $\pm$ 1.7%
Mean agreement rate	70.0%
Agreement rate std	22.1%

Appendix

Table 15: Permutation voting (MedGemma-4B,

n{=}1{,}000

K{=}4

Explore similar work

Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare
Mar 1, 2026Shravani HariprasadClinical Language Model Evaluation Prompt Sensitivity
Independent Researcher
Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
Apr 12, 2026Avi-ad Avraam BuskilaLLM Evaluation LLM Reliability
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel
Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale
Apr 26, 2026Avi-ad Avraam BuskilaRetrieval-Augmented Generation Healthcare
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

Mar 1, 2026J/K move · Enter open · S save

Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare

Shravani Hariprasad

Independent Researcher

Clinical Language Model Evaluation Prompt Sensitivity Small Language Models Clinical QA Reasoning Consistency in Language Models Instruction Following

Open paper AskPDF arXiv

Small open-source language models are gaining attention for healthcare applications in low-resource settings where cloud infrastructure and GPU hardware may be unavailable. However, the reliability of these models under different phrasings of the same clinical question remains poorly understood. We evaluate five open-source models (Gemma 2 2B, Phi-3 Mini 3.8B, Llama 3.2 3B, Mistral 7B, and Meditron-7B, a domain-pretrained model without instruction tuning) across three clinical question answering datasets (MedQA, MedMCQA, and PubMedQA) using five prompt styles: original, formal, simplified, roleplay, and direct. Model behavior is evaluated using consistency scores, accuracy, and instruction-following failure rates. All experiments were conducted locally on consumer CPU hardware without fine-tuning. Consistency and accuracy were largely independent across models. Gemma 2 achieved the highest consistency (0.845-0.888) but the lowest accuracy (33.0-43.5%), while Llama 3.2 showed moderate consistency (0.774-0.807) alongside the highest accuracy (49.0-65.0%). Roleplay prompts consistently reduced accuracy across all models, with Phi-3 Mini dropping 21.5 percentage points on MedQA. Meditron-7B exhibited near-complete instruction-following failure on PubMedQA (99.0% UNKNOWN rate), indicating that domain pretraining alone is insufficient for structured clinical question answering. These findings show that high consistency does not imply correctness: models can be reliably wrong, a dangerous failure mode in clinical AI. Llama 3.2 demonstrated the strongest balance of accuracy and reliability for low-resource deployment. Safe clinical AI requires joint evaluation of consistency, accuracy, and instruction adherence.

Abstract

Explore similar work

Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale