Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Authors: Halimat Afolabi, Zainab Afolabi, Elizabeth Friel, Jude Roberts, Antonio Ji-Xu, Lloyd Chen, Egheosa Ogbomo, Emiliomo Imevbore, Phil Eneje, Wissal El Ouahidi, Aaron Sohal, Alisa Kennan, Shreya Srivastava, Anirudh Vairavan, Laura Napitu, Katie McClure
Organizations: Stratified Precision, United Kingdom · Harvard Medical School, United States · Imperial College London, United Kingdom · National Health Service, United Kingdom · Ipsen, France · University College London, United Kingdom
Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment.
Figures & tables
Dataset
Experiment(s)
Sample Size
MedQA
Exp. 1–3 (Causal Ablation, Positional Bias, Hint Injection)
100
r/AskDocs
Exp. 4 (Human Evaluation)
30
Table 1
Figure 1: Experiments 1 (a) and 2 (b). In (b), for the unbiased and bias→gold conditions, values indicate the B-selection rate when B is the ground-truth answer. For the bias→wrong condition, values indicate the B-selection rate when B is incorrect.
Figure 2: Experiment 3 - accuracy (a) and hint adherence (b). In (b), stacked bars show the share of items that switched to the hinted option (green), switched away (red), or did not change (grey), for correct and incorrect hints. “No change” includes cases where the answer already matched the hint.
Characteristic
Laypeople (n=10)
Physicians (n=5)
Age, n (%)
18–24
6 (60)
0 (0)
25–34
3 (30)
5 (100)
35+
1 (10)
0 (0)
Gender, n (%)
Women
7 (70)
2 (40)
Table 1: Participant demographics by cohort
Figure 3: Experiment 4 - clinician–lay evaluations of model responses.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Physician survey (Experiment 4). Google Forms page displaying definitions and illustrative examples.
Figure 5: Lay participant survey (Experiment 4). Google Forms page displaying definitions and illustrative examples for each rating dimension to standardise judgments.
Model
Baseline Acc.
Ablations Acc.
Δ Acc.
ChatGPT 5
0.92 [0.85, 0.96]
0.90 [0.87, 0.93]
+0.02 [-0.04, 0.08]
Claude 4.1 Opus
0.86 [0.78, 0.91]
0.78 [0.78, 0.79]
+0.08 [0.01, 0.14]
Gemini Pro 2.5
0.89 [0.81, 0.94]
0.84 [0.81, 0.88]
+0.05 [-0.02, 0.12]
Appendix
Table 2: Experiment 1: Accuracy. Values are mean [95% CI]. Δ is Baseline − Ablations.
Model
Damage
Rescue
Casual Net Flip
Causal Density
ChatGPT 5
0.05 [0.03, 0.08]
0.31 [0.18, 0.44]
-0.28 [-0.54, -0.02]
0.10 [0.06, 0.14]
Claude 4.1 Opus
0.11 [0.09, 0.12]
0.14 [0.04, 0.24]
-0.04 [-0.16, 0.09]
0.12 [0.08, 0.17]
Gemini Pro 2.5
0.09 [0.05, 0.12]
0.22 [0.10, 0.34]
-0.16 [-0.40, 0.07]
0.11 [0.07, 0.15]
Appendix
Table 3: Experiment 1: Chain-of-thought ablation metrics. Values are mean [95% CI].
Figure 6: Accuracy remained broadly stable across conditions, indicating limited influence of positional bias. Error bars (95% Wilson CIs) are omitted here for clarity but reported in Table 4 .
Table 5: Experiment 2 - Model Predictions under Positional Bias. Δ columns are accuracy deltas vs Unbiased. Bias Net Flip (Wrong) reflects the proportion of predictions that changed to option B (biased position) in the biased to wrong condition. Acknowledgment rates are the share of hinted runs whose reasoning explicitly referenced position.
Figure 7: Experiment 2 - Model baseline accuracies vs. accuracies under positional bias. Points show mean accuracy for the unbiased condition (blue) and the biased position conditions (red): bias → gold (correct option fixed at B) and bias → wrong (incorrect option fixed at B). Numeric labels give Δ accuracy (biased − baseline); grey bars indicate the magnitude of the change.
Table 7: Experiment 3 - Flip rate relative to the unbiased condition (rate [95% CI]). A flip occurs when the prediction under the hint condition differs from the baseline prediction.
Model
Condition
n
Ack?
Accuracy
Δ vs base
Hint Adherence Rate
Claude 4.1 Opus
Hint → Gold
100
No
1.00
+0.11
1.00
Claude 4.1 Opus
Hint → Wrong
49
No
0.204
-0.633
0.796
Claude 4.1 Opus
Hint → Wrong
51
Yes
0.196
-0.745
0.804
ChatGPT 5
Hint → Gold
100
No
1.00
+0.09
1.00
ChatGPT 5
Hint → Wrong
100
No
0.26
-0.65
0.74
Gemini Pro 2.5
Hint → Gold
100
No
0.99
+0.12
0.99
Appendix
Table 8: Effect of explicit hint acknowledgment on accuracy and hint adherence. Δ column is accuracy vs unbiased; Hint Adherence Rate is the proportion of predictions matching the hinted target.
Group
k raters
Items ( n )
Item definition
ICC(2, k )
Clinicians
5
90
model × case
0.49
Laypeople
10
90
model × case
0.50
Appendix
Table 9: Panel inter-rater reliability for Exp. 4 using ICC(2, k ).
Model
Logical consistency
Medical accuracy
Completeness
Appropriateness of urgency
Potential harm
ChatGPT 5
4.10 [3.96, 4.24]
4.03 [3.88, 4.17]
4.19 [4.06, 4.32]
3.97 [3.83, 4.12]
1.69 [1.57, 1.82]
Claude 4.1 Opus
4.06 [3.95, 4.17]
3.65 [3.51, 3.80]
3.24 [3.09, 3.39]
3.50 [3.35, 3.65]
2.03 [1.89, 2.16]
Gemini Pro 2.5
4.02 [3.91, 4.13]
3.49 [3.34, 3.65]
2.98 [2.83, 3.13]
3.43 [3.28, 3.59]
2.04 [1.90, 2.18]
Appendix
Table 10: Experiment 4 - Clinician ratings per model (mean [95% CI]); higher is better except Potential harm (lower is better).
Model
Actionability
Ease of understanding
Trustworthiness
ChatGPT 5
4.36 [4.26, 4.45]
3.83 [3.71, 3.95]
3.98 [3.87, 4.09]
Claude 4.1 Opus
3.87 [3.76, 3.98]
4.03 [3.92, 4.14]
3.91 [3.80, 4.02]
Gemini Pro 2.5
3.65 [3.53, 3.76]
4.20 [4.10, 4.30]
3.83 [3.71, 3.94]
Appendix
Table 11: Experiment 4 - Layperson ratings per model (mean [95% CI]); higher is better.
Table 12: Correlations between Actionability (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.
Model
Urgency
Logic
Harm
MedAcc
Complete
Claude 4.1 Opus
0.118
0.239
0.031
-0.002
0.086
ChatGPT 5
0.160
0.409*
0.108
-0.368*
-0.377*
Gemini Pro 2.5
0.107
0.099
-0.180
0.125
-0.088
Appendix
Table 13: Correlations between Ease of understanding (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.
Model
Urgency
Logic
Harm
MedAcc
Complete
Claude 4.1 Opus
-0.020
0.109
-0.007
0.116
-0.022
ChatGPT 5
0.185
0.502**
-0.338
0.030
0.076
Gemini Pro 2.5
0.196
0.449*
-0.138
0.474**
0.223
Appendix
Table 14: Correlations between Trustworthiness (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.
Clinical Large Language Models (LLMs) achieve strong medical-exam accuracy; however, a correct answer does not guarantee that the explanation names the concepts that actually drove the decision. We introduce three lightweight, directly interpretable metrics for this faithfulness gap: the Explanation Stability Index (ESI), which measures reasoning consistency across repeated queries; the Causal Faithfulness Score (CFS), which tests whether cited concepts drive predictions via concept ablation; and the Perturbation Stability Score (PSS), which measures robustness to semantic-preserving paraphrases. By evaluating six LLMs on 150 MedQA-USMLE questions (900 model-question observations), we found that only 23.3% of the cited clinical concepts were causally necessary. Correct answers had lower CFS than incorrect answers (0.212 vs. 0.398), answer consistency negatively predicted CFS (Spearman r = -0.466), and model pairs could agree on answers while sharing only 8.8% of cited reasoning concepts. These results show that accuracy, consistency, and consensus are incomplete safety signals for clinical decision-making support. The evidence is behavioral rather than mechanistic: concept ablation tests counterfactual sensitivity of outputs, not internal circuits.
Bharath Kumar Bolla, Chaitali Deb Purkayastha, Vishnu Surya Reddy Nandi
Institute of Product Leadership, Bengaluru, India · Independent Researcher, Bengaluru, India · Amazon, Inc, Seattle, USA
Large Language Models (LLMs) are increasingly deployed in medical settings, yet their sensitivity to prompt formatting remains poorly characterized. We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions) across a broad suite of robustness tests. Our experiments reveal several concerning findings. Chain-of-Thought (CoT) prompting decreases accuracy by 5.7% compared to direct answering. Few-shot examples degrade performance by 11.9% while increasing position bias from 0.14 to 0.47. Shuffling answer options causes the model to change predictions 59.1% of the time, with accuracy dropping up to 27.4 percentage points. Front-truncating context to 50% causes accuracy to plummet below the no-context baseline, yet back-truncation preserves 97% of full-context accuracy. We further show that cloze scoring (selecting the highest log-probability option token) achieves 51.8% (4B) and 64.5% (27B), surpassing all prompting strategies and revealing that models "know" more than their generated text shows. Permutation voting recovers 4 percentage points over single-ordering inference. These results demonstrate that prompt engineering techniques validated on general-purpose models do not transfer to domain-specific medical LLMs, and that reliable alternatives exist.
Binesh Sadanandan, Vahid Behzadan
SAIL Lab, University of New Haven, West Haven, CT 06516, USA
Faithful reasoning is essential in medicine, where clinical decisions require transparent justification grounded in reliable evidence. Current medical LLMs either lack active access to evidence or use retrieved evidence without supervising how it should be appraised and applied during reasoning. To address this, we formalize evidence-based medicine principles as process-level criteria and introduce FaithMed, a framework that combines clinician-designed, automatically refined rubrics with reinforcement learning using step-level process reward assignment and advantage grouping. Across seven medical benchmarks, FaithMed improves over agentic-search baselines (+9% on average) and outcome-only RL (+5.8%), while raising average evidence-based medicine rubric scores over agentic-search Qwen3 baselines (+15.5%). This work demonstrates that explicit step-level supervision can improve both task success and the faithfulness of the reasoning process. Code is available at https://github.com/cxcscmu/FaithMed.
Zhiyun Zhang, Liwen Sun, Xiang Qian +1
Carnegie Mellon University · Stanford University School of Medicine · Xlue