Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Authors: Halimat Afolabi, Zainab Afolabi, Elizabeth Friel, Jude Roberts, Antonio Ji-Xu, Lloyd Chen, Egheosa Ogbomo, Emiliomo Imevbore, Phil Eneje, Wissal El Ouahidi, Aaron Sohal, Alisa Kennan, Shreya Srivastava, Anirudh Vairavan, Laura Napitu, Katie McClure
Organizations: Stratified Precision, United Kingdom · Harvard Medical School, United States · Imperial College London, United Kingdom · National Health Service, United Kingdom · Ipsen, France · University College London, United Kingdom
Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment.
Figures & tables
Dataset
Experiment(s)
Sample Size
MedQA
Exp. 1–3 (Causal Ablation, Positional Bias, Hint Injection)
100
r/AskDocs
Exp. 4 (Human Evaluation)
30
Table 1
Figure 1: Experiments 1 (a) and 2 (b). In (b), for the unbiased and bias→gold conditions, values indicate the B-selection rate when B is the ground-truth answer. For the bias→wrong condition, values indicate the B-selection rate when B is incorrect.
Figure 2: Experiment 3 - accuracy (a) and hint adherence (b). In (b), stacked bars show the share of items that switched to the hinted option (green), switched away (red), or did not change (grey), for correct and incorrect hints. “No change” includes cases where the answer already matched the hint.
Characteristic
Laypeople (n=10)
Physicians (n=5)
Age, n (%)
18–24
6 (60)
0 (0)
25–34
3 (30)
5 (100)
35+
1 (10)
0 (0)
Gender, n (%)
Women
7 (70)
2 (40)
Table 1: Participant demographics by cohort
Figure 3: Experiment 4 - clinician–lay evaluations of model responses.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Physician survey (Experiment 4). Google Forms page displaying definitions and illustrative examples.
Figure 5: Lay participant survey (Experiment 4). Google Forms page displaying definitions and illustrative examples for each rating dimension to standardise judgments.
Model
Baseline Acc.
Ablations Acc.
Δ Acc.
ChatGPT 5
0.92 [0.85, 0.96]
0.90 [0.87, 0.93]
+0.02 [-0.04, 0.08]
Claude 4.1 Opus
0.86 [0.78, 0.91]
0.78 [0.78, 0.79]
+0.08 [0.01, 0.14]
Gemini Pro 2.5
0.89 [0.81, 0.94]
0.84 [0.81, 0.88]
+0.05 [-0.02, 0.12]
Appendix
Table 2: Experiment 1: Accuracy. Values are mean [95% CI]. Δ is Baseline − Ablations.
Model
Damage
Rescue
Casual Net Flip
Causal Density
ChatGPT 5
0.05 [0.03, 0.08]
0.31 [0.18, 0.44]
-0.28 [-0.54, -0.02]
0.10 [0.06, 0.14]
Claude 4.1 Opus
0.11 [0.09, 0.12]
0.14 [0.04, 0.24]
-0.04 [-0.16, 0.09]
0.12 [0.08, 0.17]
Gemini Pro 2.5
0.09 [0.05, 0.12]
0.22 [0.10, 0.34]
-0.16 [-0.40, 0.07]
0.11 [0.07, 0.15]
Appendix
Table 3: Experiment 1: Chain-of-thought ablation metrics. Values are mean [95% CI].
Figure 6: Accuracy remained broadly stable across conditions, indicating limited influence of positional bias. Error bars (95% Wilson CIs) are omitted here for clarity but reported in Table 4 .
Table 5: Experiment 2 - Model Predictions under Positional Bias. Δ columns are accuracy deltas vs Unbiased. Bias Net Flip (Wrong) reflects the proportion of predictions that changed to option B (biased position) in the biased to wrong condition. Acknowledgment rates are the share of hinted runs whose reasoning explicitly referenced position.
Figure 7: Experiment 2 - Model baseline accuracies vs. accuracies under positional bias. Points show mean accuracy for the unbiased condition (blue) and the biased position conditions (red): bias → gold (correct option fixed at B) and bias → wrong (incorrect option fixed at B). Numeric labels give Δ accuracy (biased − baseline); grey bars indicate the magnitude of the change.
Table 7: Experiment 3 - Flip rate relative to the unbiased condition (rate [95% CI]). A flip occurs when the prediction under the hint condition differs from the baseline prediction.
Model
Condition
n
Ack?
Accuracy
Δ vs base
Hint Adherence Rate
Claude 4.1 Opus
Hint → Gold
100
No
1.00
+0.11
1.00
Claude 4.1 Opus
Hint → Wrong
49
No
0.204
-0.633
0.796
Claude 4.1 Opus
Hint → Wrong
51
Yes
0.196
-0.745
0.804
ChatGPT 5
Hint → Gold
100
No
1.00
+0.09
1.00
ChatGPT 5
Hint → Wrong
100
No
0.26
-0.65
0.74
Gemini Pro 2.5
Hint → Gold
100
No
0.99
+0.12
0.99
Appendix
Table 8: Effect of explicit hint acknowledgment on accuracy and hint adherence. Δ column is accuracy vs unbiased; Hint Adherence Rate is the proportion of predictions matching the hinted target.
Group
k raters
Items ( n )
Item definition
ICC(2, k )
Clinicians
5
90
model × case
0.49
Laypeople
10
90
model × case
0.50
Appendix
Table 9: Panel inter-rater reliability for Exp. 4 using ICC(2, k ).
Model
Logical consistency
Medical accuracy
Completeness
Appropriateness of urgency
Potential harm
ChatGPT 5
4.10 [3.96, 4.24]
4.03 [3.88, 4.17]
4.19 [4.06, 4.32]
3.97 [3.83, 4.12]
1.69 [1.57, 1.82]
Claude 4.1 Opus
4.06 [3.95, 4.17]
3.65 [3.51, 3.80]
3.24 [3.09, 3.39]
3.50 [3.35, 3.65]
2.03 [1.89, 2.16]
Gemini Pro 2.5
4.02 [3.91, 4.13]
3.49 [3.34, 3.65]
2.98 [2.83, 3.13]
3.43 [3.28, 3.59]
2.04 [1.90, 2.18]
Appendix
Table 10: Experiment 4 - Clinician ratings per model (mean [95% CI]); higher is better except Potential harm (lower is better).
Model
Actionability
Ease of understanding
Trustworthiness
ChatGPT 5
4.36 [4.26, 4.45]
3.83 [3.71, 3.95]
3.98 [3.87, 4.09]
Claude 4.1 Opus
3.87 [3.76, 3.98]
4.03 [3.92, 4.14]
3.91 [3.80, 4.02]
Gemini Pro 2.5
3.65 [3.53, 3.76]
4.20 [4.10, 4.30]
3.83 [3.71, 3.94]
Appendix
Table 11: Experiment 4 - Layperson ratings per model (mean [95% CI]); higher is better.
Table 12: Correlations between Actionability (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.
Model
Urgency
Logic
Harm
MedAcc
Complete
Claude 4.1 Opus
0.118
0.239
0.031
-0.002
0.086
ChatGPT 5
0.160
0.409*
0.108
-0.368*
-0.377*
Gemini Pro 2.5
0.107
0.099
-0.180
0.125
-0.088
Appendix
Table 13: Correlations between Ease of understanding (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.
Model
Urgency
Logic
Harm
MedAcc
Complete
Claude 4.1 Opus
-0.020
0.109
-0.007
0.116
-0.022
ChatGPT 5
0.185
0.502**
-0.338
0.030
0.076
Gemini Pro 2.5
0.196
0.449*
-0.138
0.474**
0.223
Appendix
Table 14: Correlations between Trustworthiness (lay) and clinician metrics by model. n=30 per model. Entries are r; * p<.05, ** p<.01, *** p<.001.