Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Abinitha Gourabathina; Haoran Zhang; Yuexing Hao; Walter Gerych; Marzyeh Ghassemi

cs.AISep 29, 2026

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Authors: Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi

Organizations: Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science

Abstract

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.

Figures & tables

	Corpus		Model	Physician
Dataset	Case	Scen.	Resp.	Case	Scen.	Annot.
AskDocs	195	888	39,723	165	822	2,468
OncQA	100	377	16,784	58	290	878
SCT	148	442	19,890	148	442	1,326
USMLE Derm	1,511	4,525	149,794	300	900	2,684
Total	1,954	6,232	226,191	671	2,454	7,356

Table 1: Dataset composition and evaluation coverage. A) Number of cases and scenarios in each source dataset, together with the corresponding model responses and physician annotations collected for evaluation. B) Distribution of scenarios across the baseline and perturbation conditions. We define a case as a unique underlying clinical example and a scenario as a particular presentation of that case, either in its baseline form or under one perturbation. Tone perturbations are applied only to AskDocs and OncQA.

Figure 1: Examples of gender and tone perturbations from our benchmark.

Figure 2: Baseline accuracy across clinical management tasks. Accuracy of physician and LLM recommendations on baseline (unperturbed) clinical scenarios across the three binary management decisions: self-management ( Manage ), in-person clinical visit ( Visit ), and medical resource allocation ( Resource ). Accuracies are calculated with 95% Wilson CIs using baseline physician majority vote.

Figure 3: Treatment recommendation tradeoffs for clinicians and LLMs. We compare harm rate and overburden rate across the three clinical decision tasks. Points closer to the lower-left corner reflect more desirable behavior, with fewer missed necessary interventions and fewer unnecessary escalations.

Figure 4: Accuracy degradation from perturbation. ( Top ) Baseline accuracy averaged across the three clinical recommendation tasks. ( Bottom ) accuracy drop under perturbation relative to each rater’s baseline performance with 95% CIs. Accuracy is computed against gold-standard labels derived from clinician majority vote on the baseline scenario. Negative values indicate performance degradation.

Rater / Model	Manage	Visit	Resource	Overall
Clinician (avg)	$+0.5$	$+0.5$	$+1.4$	$+0.8$
GPT-4o	$-2.2$	$-6.0$	$-2.6$	$-3.6$
DeepSeek-R1-32B	$-0.7$	$-12.5$	$-19.5$	$-14.2$
MedGemma-27B	$-3.7$	$-0.2$	$+0.1$	$-1.3$
Llama-3.3-70B	$-4.6$	$-2.5$	$-3.4$	$-3.5$
Qwen2.5-32B	$-6.9$	$-16.3$	$-15.6$	$-13.0$

Table 2: Sensitivity on clinician-stable cases. Mean paired change in accuracy (percentage points) from the matched baseline to the perturbed scenario, averaged across the four perturbations. Evaluation is restricted to cases where the physician majority recommendation does not change between baseline and perturbation. Negative values indicate degradation.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Input Type	Sample Input
AskDocs	Reddit post (patient-authored, demographics + free-text)	28F, ex-smoker, no drinking, 5 ′ 3 ′′ , 200 lbs. My arms sometimes hurt when I sneeze. It’s not always, just sometimes I get a somewhat intense ache down either or both arms right after I sneeze. Should I be worried?!
OncQA	EHR summary + patient message	EHR Context: 47-year-old female. Cancer: Stage II invasive ductal carcinoma. PMH: asthma, obesity. Treatments: lumpectomy (2 months ago); adjuvant doxorubicin/cyclophosphamide (started 1 month ago). Most recent visit: tolerating treatment well; continue as planned. Patient message: I’ve noticed that my hair has started falling out more than usual. Is this a side effect of my treatment? What can I do to minimize hair loss?
SCT	Clinical vignette with action–finding pair	A 66-year-old male is 1 day post-op following an elective total hip replacement. During the night he complains of shortness of breath. PMH: COPD, heart failure. RR 24; O 2 sat 93% on 10 L/min via non-rebreather mask. If you were thinking of: Ordering a blood culture And then you find: Diffuse coarse crackles in both lungs
USMLE	Multiple-choice clinical vignette	A 19-year-old woman presents with a 1-month history of mild fatigue and weakness. Physical exam shows no abnormalities. Hemoglobin 11 g/dL; MCV 74 $\mu$ m 3 ; hemoglobin electrophoresis: HbA2 10% (normal $<$ 3.5%). Options: (A) Beta thalassemia minor (B) Alpha thalassemia minima (C) Hemoglobin Barts disease (D) Hemoglobin H disease
Derm	Multiple-choice clinical vignette (image-paired)	A 70-year-old man presents with a 1-year history of a non-healing growth on the right cheek. He first noticed it when he accidentally bumped it and it bled easily. He denies any preceding trauma to the lesion. Options: (A) Squamous cell carcinoma (B) Melanoma (C) Atypical fibroxanthoma (D) Basal cell carcinoma

Appendix

Table 3: Representative inputs from each source dataset (baseline, unperturbed). AskDocs : de-identified Reddit medical Q&A posts with self-reported demographics. OncQA : structured EHR summaries paired with patient-authored messages to oncology teams. SCT : Script Concordance Test items presenting a clinical vignette alongside a candidate management action and a new finding, requiring raters to judge whether the finding increases, decreases, or does not change the appropriateness of the action. USMLE : MedQA multiple-choice questions from United States Medical Licensing Examination Step 1/2. Derm : dermatology multiple-choice vignettes.

Category	Terms
Pregnancy & reproductive anatomy	pregnant, pregnancy, gestation, gestational, maternal, fetal, fetus, prenatal, antenatal, postnatal, postpartum, lactation, breastfeeding, contraception, contraceptive, birth control, conception, ovulation, fertility, infertility, miscarriage, abortion, stillbirth, trimester, amniocentesis, ultrasound, prenatal care, maternal health, prenatal screening
Sex-specific genitalia	penis, vagina, vulva, clitoris, testicle, testis, scrotum, ovary, ovaries, uterus, cervix, fallopian, prostate, genital, genitalia, reproductive, reproduction, labia, mons pubis, epididymis, vas deferens, seminal vesicle, bulbourethral, bartholin
Gendered conditions	menopause, menstruation, menstrual, period, menses, dysmenorrhea, amenorrhea, menorrhagia, endometriosis, fibroids, uterine, uterine fibroids, ovarian, cervical, prostate, testicular, breast, mammary, gynecological, gynecology, urological, urology, andrology, premenstrual, pms, dyspareunia, vaginismus, vulvodynia, erectile dysfunction, premature ejaculation, low testosterone
Gender-specific cancers	ovarian cancer, cervical cancer, uterine cancer, endometrial cancer, prostate cancer, testicular cancer, breast cancer, penile cancer, vulvar cancer, vaginal cancer, ovarian carcinoma, cervical carcinoma, endometrial carcinoma, prostate carcinoma, testicular carcinoma, breast carcinoma, penile carcinoma, vulvar carcinoma, vaginal carcinoma
Hormonal conditions	estrogen, progesterone, testosterone, hormone, hormonal, pcos, polycystic ovary, hirsutism, androgen, androgenic, hormone therapy, hormone replacement, hrt, birth control pill, oral contraceptive, hormonal imbalance, thyroid, thyroid hormone
Reproductive health procedures	fertility treatment, ivf, in vitro fertilization, artificial insemination, sperm donation, egg donation, surrogacy, tubal ligation, vasectomy, hysterectomy, oophorectomy, orchiectomy, mastectomy, lumpectomy

Appendix

Table 4: Gender-relevant terms used for filtering. Terms are grouped by clinical category.

Figure 5: Baseline clinical decision-making performance across three tasks ( Manage, Visit, Resource ) for clinicians and five LLMs, measured by accuracy, Balanced Accuracy, and Cohen’s $\kappa$ . This figure serves as a control, establishing pre-perturbation performance before any stylistic modifications are applied to clinical contexts.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	85.4 [83.7, 86.8]	85.2 [82.4, 87.7]	70.6 [66.5, 74.4]	77.8 [73.9, 81.2]	90.7 [88.2, 92.7]	84.4 [81.1, 87.1]
Gender Swap	83.0 [81.3, 84.6]	87.2 [84.5, 89.6]	69.6 [65.7, 73.3]	73.7 [69.9, 77.2]	87.1 [84.4, 89.5]	81.2 [77.9, 84.2]
Gender Removal	83.5 [81.8, 85.0]	86.7 [84.0, 89.1]	64.8 [61.1, 68.3]	73.1 [69.6, 76.4]	88.2 [85.5, 90.4]	79.6 [76.4, 82.5]
Colorful Tone	78.5 [75.3, 81.5]	80.5 [74.8, 85.2]	65.2 [58.7, 71.1]	65.8 [59.3, 71.7]	82.2 [76.6, 86.7]	65.6 [59.1, 71.6]
Uncertain Tone	80.9 [77.7, 83.7]	81.6 [76.0, 86.1]	68.0 [61.6, 73.8]	70.7 [64.4, 76.3]	80.4 [74.6, 85.1]	70.7 [64.4, 76.3]

Appendix

Table 5: Accuracy on the MANAGE task (recommending self-care vs. clinical management) across perturbation conditions and raters. Values are accuracy (%) with 95% Wilson confidence intervals in brackets. Clinician accuracy is pooled across all individual rater annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	94.0 [92.8, 94.9]	96.4 [94.7, 97.6]	82.8 [79.2, 85.9]	70.1 [65.9, 73.9]	89.5 [86.8, 91.6]	92.7 [90.3, 94.6]
Gender Swap	91.7 [90.4, 92.8]	89.6 [87.1, 91.7]	59.6 [55.5, 63.6]	69.8 [65.8, 73.5]	87.1 [84.4, 89.5]	79.1 [75.7, 82.2]
Gender Removal	91.6 [90.3, 92.7]	88.1 [85.4, 90.3]	59.5 [55.8, 63.2]	65.9 [62.2, 69.4]	88.6 [86.0, 90.8]	76.8 [73.4, 79.8]
Colorful Tone	87.8 [85.1, 90.1]	80.5 [74.8, 85.2]	61.8 [55.2, 68.0]	71.0 [64.7, 76.6]	83.6 [78.1, 87.9]	74.7 [68.5, 79.9]
Uncertain Tone	90.7 [88.3, 92.7]	81.6 [76.0, 86.1]	59.0 [52.4, 65.3]	72.5 [66.3, 78.0]	82.2 [76.6, 86.7]	71.2 [64.9, 76.7]

Appendix

Table 6: Accuracy on the VISIT task (visiting a clinician) across perturbation conditions and raters. Values are accuracy (%) with 95% Wilson confidence intervals in brackets. Clinician accuracy is pooled across all individual rater annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	92.1 [90.8, 93.2]	93.4 [91.3, 95.1]	80.7 [77.0, 83.9]	74.4 [70.4, 78.1]	89.3 [86.7, 91.5]	92.5 [90.1, 94.4]
Gender Swap	89.9 [88.5, 91.2]	89.3 [86.8, 91.5]	61.6 [57.5, 65.6]	72.1 [68.2, 75.7]	87.3 [84.5, 89.6]	80.7 [77.4, 83.7]
Gender Removal	89.9 [88.5, 91.1]	88.5 [85.9, 90.7]	62.7 [59.0, 66.3]	70.1 [66.5, 73.4]	87.4 [84.7, 89.7]	79.5 [76.2, 82.4]
Colorful Tone	84.5 [81.6, 87.1]	78.7 [72.9, 83.6]	61.4 [54.8, 67.6]	73.8 [67.6, 79.1]	81.3 [75.6, 85.9]	72.4 [66.2, 77.9]
Uncertain Tone	86.6 [83.8, 88.9]	81.6 [76.0, 86.1]	60.4 [53.8, 66.6]	76.6 [70.6, 81.7]	80.4 [74.6, 85.1]	73.9 [67.7, 79.2]

Appendix

Table 7: Accuracy on the RESOURCE task (ordering tests / referrals) across perturbation conditions and raters. Values are accuracy (%) with 95% Wilson confidence intervals in brackets. Clinician accuracy is pooled across all individual rater annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	0.837 [0.813, 0.861]	0.816 [0.772, 0.859]	0.619 [0.563, 0.674]	0.627 [0.576, 0.680]	0.739 [0.690, 0.787]	0.830 [0.787, 0.871]
Gender Swap	0.783 [0.756, 0.810]	0.773 [0.726, 0.821]	0.587 [0.534, 0.640]	0.609 [0.558, 0.660]	0.664 [0.619, 0.712]	0.764 [0.717, 0.810]
Gender Removal	0.805 [0.780, 0.829]	0.770 [0.721, 0.818]	0.525 [0.475, 0.574]	0.593 [0.543, 0.645]	0.687 [0.638, 0.735]	0.763 [0.715, 0.808]
Colorful Tone	0.750 [0.711, 0.787]	0.742 [0.677, 0.806]	0.571 [0.502, 0.641]	0.520 [0.462, 0.578]	0.734 [0.672, 0.796]	0.672 [0.605, 0.740]
Uncertain Tone	0.820 [0.789, 0.850]	0.788 [0.725, 0.849]	0.605 [0.531, 0.675]	0.604 [0.539, 0.671]	0.707 [0.645, 0.767]	0.713 [0.644, 0.776]

Appendix

Table 8: Balanced Accuracy on the Manage task (recommending self-care vs. clinical management) across perturbation conditions and raters. Balanced Accuracy is computed as

(\text{TPR}+\text{TNR})/2

; 95% bootstrap confidence intervals shown in brackets. Clinician values are pooled across all individual rater annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	0.925 [0.905, 0.941]	0.953 [0.927, 0.976]	0.809 [0.761, 0.855]	0.682 [0.623, 0.738]	0.686 [0.639, 0.734]	0.905 [0.869, 0.940]
Gender Swap	0.870 [0.845, 0.894]	0.824 [0.775, 0.866]	0.539 [0.482, 0.596]	0.666 [0.613, 0.721]	0.621 [0.578, 0.668]	0.751 [0.700, 0.801]
Gender Removal	0.889 [0.868, 0.909]	0.823 [0.779, 0.869]	0.511 [0.458, 0.563]	0.621 [0.569, 0.671]	0.651 [0.605, 0.700]	0.736 [0.687, 0.782]
Colorful Tone	0.838 [0.803, 0.873]	0.736 [0.666, 0.805]	0.594 [0.517, 0.672]	0.645 [0.573, 0.723]	0.723 [0.654, 0.790]	0.749 [0.683, 0.815]
Uncertain Tone	0.882 [0.851, 0.912]	0.772 [0.704, 0.837]	0.572 [0.497, 0.647]	0.693 [0.620, 0.761]	0.707 [0.640, 0.774]	0.722 [0.653, 0.790]

Appendix

Table 9: Balanced Accuracy on the Visit task (visiting a clinician) across perturbation conditions and raters. Balanced Accuracy is computed as

(\text{TPR}+\text{TNR})/2

; 95% bootstrap confidence intervals shown in brackets. Clinician values are pooled across all individual rater annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Baseline	0.907 [0.888, 0.924]	0.931 [0.906, 0.956]	0.781 [0.732, 0.827]	0.681 [0.625, 0.732]	0.697 [0.650, 0.744]	0.902 [0.865, 0.936]
Gender Swap	0.867 [0.845, 0.888]	0.844 [0.801, 0.883]	0.534 [0.481, 0.586]	0.672 [0.620, 0.723]	0.647 [0.604, 0.692]	0.750 [0.701, 0.798]
Gender Removal	0.873 [0.851, 0.894]	0.835 [0.793, 0.878]	0.559 [0.509, 0.610]	0.618 [0.567, 0.668]	0.652 [0.607, 0.694]	0.741 [0.693, 0.788]
Colorful Tone	0.814 [0.781, 0.847]	0.734 [0.666, 0.798]	0.569 [0.499, 0.643]	0.669 [0.606, 0.734]	0.698 [0.638, 0.760]	0.715 [0.647, 0.783]
Uncertain Tone	0.871 [0.844, 0.897]	0.805 [0.741, 0.862]	0.589 [0.519, 0.660]	0.698 [0.634, 0.764]	0.687 [0.628, 0.752]	0.707 [0.633, 0.775]

Appendix

Table 10: Balanced Accuracy on the Resource task (ordering tests / referrals) across perturbation conditions and raters. Balanced Accuracy is computed as

(\text{TPR}+\text{TNR})/2

; 95% bootstrap confidence intervals shown in brackets. Clinician values are pooled across all individual rater annotations.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	84.8 [81.4,87.7]	84.2 [77.9,89.0]	60.1 [52.5,67.3]	69.5 [62.1,76.0]	88.5 [82.7,92.5]	75.6 [68.5,81.5]
	Visit	90.9 [88.0,93.1]	94.5 [90.0,97.1]	81.0 [74.3,86.3]	68.3 [60.8,74.9]	85.5 [79.3,90.0]	91.5 [86.2,94.8]
	Resource	88.7 [85.6,91.2]	90.9 [85.5,94.4]	78.5 [71.6,84.1]	70.7 [63.4,77.2]	84.2 [77.9,89.0]	90.2 [84.7,93.9]
	Avg	88.1 [84.6,91.6]	89.9 [84.0,95.8]	73.2 [60.3,86.1]	69.5 [68.1,70.9]	86.1 [83.6,88.5]	85.8 [75.8,95.8]
OncQA	Manage	82.0 [75.7,87.0]	87.9 [77.1,94.0]	81.0 [69.1,89.1]	86.2 [75.1,92.8]	89.7 [79.2,95.2]	82.8 [71.1,90.4]
	Visit	95.5 [91.4,97.7]	94.8 [85.9,98.2]	84.5 [73.1,91.6]	82.8 [71.1,90.4]	91.4 [81.4,96.3]	91.4 [81.4,96.3]

Appendix

Table 11: Accuracy (%) under the Baseline condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	77.9 [74.1,81.4]	82.2 [75.6,87.3]	62.2 [54.6,69.3]	68.5 [61.0,75.1]	81.2 [74.6,86.4]	74.4 [67.2,80.5]
	Visit	88.5 [85.3,91.0]	86.5 [80.4,90.9]	51.8 [44.2,59.3]	68.5 [61.0,75.1]	77.0 [70.0,82.7]	70.1 [62.7,76.6]
	Resource	84.8 [81.4,87.7]	84.7 [78.3,89.4]	59.8 [52.1,67.0]	73.9 [66.8,80.0]	76.4 [69.3,82.2]	73.2 [65.9,79.4]
	Avg	83.7 [77.7,89.8]	84.5 [82.0,86.9]	57.9 [51.8,64.1]	70.3 [66.7,73.9]	78.2 [75.2,81.2]	72.6 [70.1,75.0]
OncQA	Manage	74.3 [67.3,80.2]	84.2 [72.6,91.5]	77.6 [65.3,86.4]	86.2 [75.1,92.8]	81.0 [69.1,89.1]	81.0 [69.1,89.1]
	Visit	94.3 [89.8,96.9]	89.5 [78.9,95.1]	77.6 [65.3,86.4]	82.8 [71.1,90.4]	89.7 [79.2,95.2]	81.0 [69.1,89.1]

Appendix

Table 12: Accuracy (%) under the Gender Swap condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	80.6 [76.9,83.8]	81.8 [75.2,87.0]	49.7 [42.2,57.2]	63.0 [55.4,70.0]	82.4 [75.9,87.5]	74.5 [67.4,80.6]
	Visit	86.5 [83.2,89.2]	82.4 [75.9,87.5]	56.4 [48.7,63.7]	70.3 [62.9,76.7]	81.8 [75.2,87.0]	71.5 [64.2,77.9]
	Resource	83.4 [79.9,86.4]	84.2 [77.9,89.0]	60.0 [52.4,67.2]	69.7 [62.3,76.2]	77.0 [70.0,82.7]	73.9 [66.8,80.0]
	Avg	83.5 [80.2,86.8]	82.8 [81.4,84.3]	55.4 [49.4,61.3]	67.7 [63.1,72.2]	80.4 [77.0,83.8]	73.3 [71.5,75.1]
OncQA	Manage	73.0 [65.9,79.0]	70.7 [58.0,80.8]	81.0 [69.1,89.1]	87.9 [77.1,94.0]	87.7 [76.8,93.9]	77.6 [65.3,86.4]
	Visit	90.2 [84.9,93.8]	84.5 [73.1,91.6]	75.9 [63.5,85.0]	81.0 [69.1,89.1]	93.0 [83.3,97.2]	81.0 [69.1,89.1]

Appendix

Table 13: Accuracy (%) under the Gender Removal condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	80.2 [76.4,83.5]	81.6 [74.9,86.8]	60.4 [52.7,67.5]	58.5 [50.9,65.8]	82.4 [75.9,87.5]	68.1 [60.6,74.8]
	Visit	85.7 [82.3,88.5]	78.5 [71.6,84.1]	62.0 [54.3,69.1]	65.9 [58.3,72.7]	81.8 [75.2,87.0]	74.8 [67.7,80.9]
	Resource	83.5 [79.9,86.5]	79.1 [72.3,84.7]	61.3 [53.7,68.5]	68.3 [60.8,74.9]	78.2 [71.3,83.8]	71.2 [63.8,77.6]
	Avg	83.1 [80.0,86.3]	79.8 [77.9,81.6]	61.2 [60.3,62.1]	64.2 [58.5,70.0]	80.8 [78.2,83.4]	71.4 [67.5,75.2]
OncQA	Manage	73.9 [66.9,79.8]	77.6 [65.3,86.4]	78.9 [66.7,87.5]	86.2 [75.1,92.8]	81.5 [69.2,89.6]	58.6 [45.8,70.4]
	Visit	93.8 [89.2,96.5]	86.2 [75.1,92.8]	61.4 [48.4,72.9]	86.0 [74.7,92.7]	88.9 [77.8,94.8]	74.1 [61.6,83.7]

Appendix

Table 14: Accuracy (%) under the Colorful Tone condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means. Only AskDocs and OncQA include tone-perturbed annotations.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	81.0 [77.3,84.2]	83.6 [77.2,88.5]	65.5 [57.9,72.3]	68.5 [61.0,75.1]	80.0 [73.2,85.4]	73.3 [66.1,79.5]
	Visit	89.7 [86.7,92.1]	81.2 [74.6,86.4]	58.2 [50.6,65.4]	64.8 [57.3,71.7]	79.4 [72.6,84.9]	72.1 [64.8,78.4]
	Resource	84.8 [81.4,87.7]	80.0 [73.2,85.4]	60.6 [53.0,67.7]	70.3 [62.9,76.7]	77.0 [70.0,82.7]	72.1 [64.8,78.4]
	Avg	85.2 [80.3,90.1]	81.6 [79.5,83.7]	61.4 [57.2,65.6]	67.9 [64.7,71.0]	78.8 [77.0,80.6]	72.5 [71.7,73.3]
OncQA	Manage	80.6 [74.1,85.8]	75.9 [63.5,85.0]	75.4 [62.9,84.8]	77.2 [64.8,86.2]	81.5 [69.2,89.6]	63.2 [50.2,74.5]
	Visit	93.7 [89.1,96.5]	82.8 [71.1,90.4]	61.4 [48.4,72.9]	94.7 [85.6,98.2]	90.7 [80.1,96.0]	68.4 [55.5,79.0]

Appendix

Table 15: Accuracy (%) under the Uncertain Tone condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means. Only AskDocs and OncQA include tone-perturbed annotations.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	84.8 [81.4,87.7]	84.2 [77.9,89.0]	60.1 [52.5,67.3]	69.5 [62.1,76.0]	88.5 [82.7,92.5]	75.6 [68.5,81.5]
	Visit	90.9 [88.0,93.1]	94.5 [90.0,97.1]	81.0 [74.3,86.3]	68.3 [60.8,74.9]	85.5 [79.3,90.0]	91.5 [86.2,94.8]
	Resource	88.7 [85.6,91.2]	90.9 [85.5,94.4]	78.5 [71.6,84.1]	70.7 [63.4,77.2]	84.2 [77.9,89.0]	90.2 [84.7,93.9]
	Avg	88.1 [84.6,91.6]	89.9 [84.0,95.8]	73.2 [60.3,86.1]	69.5 [68.1,70.9]	86.1 [83.6,88.5]	85.8 [75.8,95.8]
OncQA	Manage	82.0 [75.7,87.0]	87.9 [77.1,94.0]	81.0 [69.1,89.1]	86.2 [75.1,92.8]	89.7 [79.2,95.2]	82.8 [71.1,90.4]
	Visit	95.5 [91.4,97.7]	94.8 [85.9,98.2]	84.5 [73.1,91.6]	82.8 [71.1,90.4]	91.4 [81.4,96.3]	91.4 [81.4,96.3]

Appendix

Table 16: Accuracy (%) under the Baseline condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	77.9 [74.1,81.4]	82.2 [75.6,87.3]	62.2 [54.6,69.3]	68.5 [61.0,75.1]	81.2 [74.6,86.4]	74.4 [67.2,80.5]
	Visit	88.5 [85.3,91.0]	86.5 [80.4,90.9]	51.8 [44.2,59.3]	68.5 [61.0,75.1]	77.0 [70.0,82.7]	70.1 [62.7,76.6]
	Resource	84.8 [81.4,87.7]	84.7 [78.3,89.4]	59.8 [52.1,67.0]	73.9 [66.8,80.0]	76.4 [69.3,82.2]	73.2 [65.9,79.4]
	Avg	83.7 [77.7,89.8]	84.5 [82.0,86.9]	57.9 [51.8,64.1]	70.3 [66.7,73.9]	78.2 [75.2,81.2]	72.6 [70.1,75.0]
OncQA	Manage	74.3 [67.3,80.2]	84.2 [72.6,91.5]	77.6 [65.3,86.4]	86.2 [75.1,92.8]	81.0 [69.1,89.1]	81.0 [69.1,89.1]
	Visit	94.3 [89.8,96.9]	89.5 [78.9,95.1]	77.6 [65.3,86.4]	82.8 [71.1,90.4]	89.7 [79.2,95.2]	81.0 [69.1,89.1]

Appendix

Table 17: Accuracy (%) under the Gender Swap condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	80.6 [76.9,83.8]	81.8 [75.2,87.0]	49.7 [42.2,57.2]	63.0 [55.4,70.0]	82.4 [75.9,87.5]	74.5 [67.4,80.6]
	Visit	86.5 [83.2,89.2]	82.4 [75.9,87.5]	56.4 [48.7,63.7]	70.3 [62.9,76.7]	81.8 [75.2,87.0]	71.5 [64.2,77.9]
	Resource	83.4 [79.9,86.4]	84.2 [77.9,89.0]	60.0 [52.4,67.2]	69.7 [62.3,76.2]	77.0 [70.0,82.7]	73.9 [66.8,80.0]
	Avg	83.5 [80.2,86.8]	82.8 [81.4,84.3]	55.4 [49.4,61.3]	67.7 [63.1,72.2]	80.4 [77.0,83.8]	73.3 [71.5,75.1]
OncQA	Manage	73.0 [65.9,79.0]	70.7 [58.0,80.8]	81.0 [69.1,89.1]	87.9 [77.1,94.0]	87.7 [76.8,93.9]	77.6 [65.3,86.4]
	Visit	90.2 [84.9,93.8]	84.5 [73.1,91.6]	75.9 [63.5,85.0]	81.0 [69.1,89.1]	93.0 [83.3,97.2]	81.0 [69.1,89.1]

Appendix

Table 18: Accuracy (%) under the Gender Removal condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	80.2 [76.4,83.5]	81.6 [74.9,86.8]	60.4 [52.7,67.5]	58.5 [50.9,65.8]	82.4 [75.9,87.5]	68.1 [60.6,74.8]
	Visit	85.7 [82.3,88.5]	78.5 [71.6,84.1]	62.0 [54.3,69.1]	65.9 [58.3,72.7]	81.8 [75.2,87.0]	74.8 [67.7,80.9]
	Resource	83.5 [79.9,86.5]	79.1 [72.3,84.7]	61.3 [53.7,68.5]	68.3 [60.8,74.9]	78.2 [71.3,83.8]	71.2 [63.8,77.6]
	Avg	83.1 [80.0,86.3]	79.8 [77.9,81.6]	61.2 [60.3,62.1]	64.2 [58.5,70.0]	80.8 [78.2,83.4]	71.4 [67.5,75.2]
OncQA	Manage	73.9 [66.9,79.8]	77.6 [65.3,86.4]	78.9 [66.7,87.5]	86.2 [75.1,92.8]	81.5 [69.2,89.6]	58.6 [45.8,70.4]
	Visit	93.8 [89.2,96.5]	86.2 [75.1,92.8]	61.4 [48.4,72.9]	86.0 [74.7,92.7]	88.9 [77.8,94.8]	74.1 [61.6,83.7]

Appendix

Table 19: Accuracy (%) under the Colorful Tone condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means. Only AskDocs and OncQA include tone-perturbed annotations.

Dataset	Task	Clin.	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
AskDocs	Manage	81.0 [77.3,84.2]	83.6 [77.2,88.5]	65.5 [57.9,72.3]	68.5 [61.0,75.1]	80.0 [73.2,85.4]	73.3 [66.1,79.5]
	Visit	89.7 [86.7,92.1]	81.2 [74.6,86.4]	58.2 [50.6,65.4]	64.8 [57.3,71.7]	79.4 [72.6,84.9]	72.1 [64.8,78.4]
	Resource	84.8 [81.4,87.7]	80.0 [73.2,85.4]	60.6 [53.0,67.7]	70.3 [62.9,76.7]	77.0 [70.0,82.7]	72.1 [64.8,78.4]
	Avg	85.2 [80.3,90.1]	81.6 [79.5,83.7]	61.4 [57.2,65.6]	67.9 [64.7,71.0]	78.8 [77.0,80.6]	72.5 [71.7,73.3]
OncQA	Manage	80.6 [74.1,85.8]	75.9 [63.5,85.0]	75.4 [62.9,84.8]	77.2 [64.8,86.2]	81.5 [69.2,89.6]	63.2 [50.2,74.5]
	Visit	93.7 [89.1,96.5]	82.8 [71.1,90.4]	61.4 [48.4,72.9]	94.7 [85.6,98.2]	90.7 [80.1,96.0]	68.4 [55.5,79.0]

Appendix

Table 20: Accuracy (%) under the Uncertain Tone condition by dataset and task. Per-task values ( Manage , Visit , Resource ) and their macro mean ( Avg ) are shown for each rater. The Average block aggregates across datasets. 95% Wilson CIs in brackets; Average-block CIs over dataset means. Only AskDocs and OncQA include tone-perturbed annotations.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Gender Swap ( $n$ =515)
Baseline (matched)	92.1 [90.7, 93.4]	92.8 [90.3, 94.7]	71.2 [66.5, 75.6]	77.6 [73.1, 81.5]	92.1 [89.4, 94.2]	85.3 [81.6, 88.3]
Perturbed	92.8 [91.4, 94.0]	92.8 [90.2, 94.7]	71.2 [66.7, 75.3]	74.7 [70.4, 78.6]	88.3 [85.3, 90.8]	82.3 [78.5, 85.5]
Gender Removal ( $n$ =519)
Baseline (matched)	92.8 [91.4, 94.0]	94.2 [91.9, 95.9]	71.2 [66.5, 75.5]	79.9 [75.7, 83.6]	91.3 [88.5, 93.5]	86.5 [83.0, 89.4]
Perturbed	92.7 [91.3, 93.9]	92.7 [90.1, 94.6]	65.2 [61.0, 69.2]	73.6 [69.7, 77.3]	89.2 [86.2, 91.6]	81.6 [78.1, 84.7]

Appendix

Table 21: Accuracy on the Manage task (recommending self-care vs. clinical management) restricted to contexts where the clinician majority verdict is stable across perturbation conditions (i.e., the majority vote does not flip relative to baseline).

n

denotes the number of annotated contexts retained per condition. Within each block the matched baseline is computed over exactly those same contexts, so the two rows are paired and the difference isolates the perturbation. 95% Wilson confidence intervals in brackets.

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Gender Swap ( $n$ =605)
Baseline (matched)	96.5 [95.5, 97.2]	98.2 [96.8, 99.0]	83.9 [80.2, 87.0]	70.5 [66.2, 74.6]	90.7 [88.0, 92.8]	94.5 [92.1, 96.1]
Perturbed	96.4 [95.5, 97.2]	95.2 [93.2, 96.6]	60.8 [56.4, 65.0]	71.9 [67.8, 75.7]	88.7 [86.0, 91.0]	80.7 [77.1, 83.8]
Gender Removal ( $n$ =603)
Baseline (matched)	96.5 [95.5, 97.2]	97.8 [96.3, 98.7]	84.2 [80.5, 87.2]	71.8 [67.5, 75.8]	91.0 [88.4, 93.1]	94.7 [92.3, 96.3]
Perturbed	96.8 [95.9, 97.5]	94.4 [92.2, 95.9]	60.5 [56.5, 64.3]	66.3 [62.4, 70.0]	90.2 [87.5, 92.3]	78.8 [75.3, 81.9]

Appendix

Table 22: Accuracy on the Visit task (visiting a clinician) restricted to contexts where the clinician majority verdict is stable across perturbation conditions (i.e., the majority vote does not flip relative to baseline).

n

Perturbation	Clinician	GPT-4o	DeepSeek	MedGemma	Llama	Qwen
Gender Swap ( $n$ =598)
Baseline (matched)	94.7 [93.6, 95.7]	95.5 [93.5, 96.9]	80.0 [76.0, 83.4]	74.5 [70.2, 78.3]	90.1 [87.4, 92.3]	92.8 [90.2, 94.7]
Perturbed	94.8 [93.7, 95.7]	94.3 [92.1, 95.9]	61.3 [56.9, 65.5]	73.6 [69.5, 77.3]	88.1 [85.3, 90.5]	81.1 [77.6, 84.2]
Gender Removal ( $n$ =594)
Baseline (matched)	94.9 [93.8, 95.8]	95.5 [93.5, 96.9]	81.2 [77.3, 84.6]	74.6 [70.3, 78.5]	90.2 [87.5, 92.4]	94.3 [91.9, 96.0]
Perturbed	95.7 [94.6, 96.5]	94.1 [91.9, 95.7]	63.4 [59.5, 67.2]	70.2 [66.4, 73.8]	88.7 [85.9, 91.0]	81.0 [77.7, 84.0]

Appendix

Table 23: Accuracy on the Resource task (ordering tests / referrals) restricted to contexts where the clinician majority verdict is stable across perturbation conditions (i.e., the majority vote does not flip relative to baseline).

n

Condition	% Agree	Kripp. $\alpha$	Fleiss’ $\kappa$
Baseline	92.7	0.843	0.842
Gender Swap	92.8	0.849	0.850
Gender Removal	93.6	0.865	0.865
Colorful Tone	90.5	0.805	0.804
Uncertain Tone	91.4	0.825	0.826

Appendix

Table 24: Physician inter-annotator agreement by perturbation condition (all datasets and tasks pooled).

Condition	Task	% Agree	$\alpha$	$\kappa$
Baseline	Manage	91	0.67	0.67
Baseline	Visit	95	0.79	0.79
Baseline	Resource	93	0.73	0.73
Gender Swap	Manage	91	0.68	0.69
Gender Swap	Visit	95	0.78	0.78
Gender Swap	Resource	93	0.75	0.74

Appendix

Table 25: Physician inter-annotator agreement by perturbation condition and task.

Dataset	Condition	% Agree	$\alpha$	$\kappa$
AskDocs	Baseline	85.9	0.711	0.707
AskDocs	Gender Swap	85.3	0.705	0.706
AskDocs	Gender Removal	88.6	0.769	0.769
AskDocs	Colorful Tone	91.3	0.825	0.826
AskDocs	Uncertain Tone	92.9	0.856	0.856
OncQA	Baseline	89.4	0.749	0.745

Appendix

Table 26: Physician inter-annotator agreement by source dataset and perturbation condition. Tone perturbations exist only for AskDocs and OncQA.

	Manage		Visit		Resource
Rater / Model	Base	Swap	Base	Swap	Base	Swap
Physician (majority)	$+0.0$ [ $+0.0$ , $+0.0$ ]	$+0.7$ [ $-2.9$ , $+4.4$ ]	$+0.0$ [ $+0.0$ , $+0.0$ ]	$+0.3$ [ $-2.2$ , $+2.7$ ]	$+0.0$ [ $+0.0$ , $+0.0$ ]	$-1.4$ [ $-4.6$ , $+1.9$ ]
Physician (per rater)	$+0.0$ [ $-2.3$ , $+2.3$ ]	$-0.4$ [ $-3.1$ , $+2.3$ ]	$-0.1$ [ $-1.7$ , $+1.6$ ]	$-0.2$ [ $-2.2$ , $+1.7$ ]	$\mathbf{-2.5}$ [ $-4.5$ , $-0.5$ ]	$-2.2$ [ $-4.6$ , $+0.2$ ]
GPT-4o	$-0.3$ [ $-2.2$ , $+1.6$ ]	$+0.8$ [ $-1.9$ , $+3.4$ ]	$+1.1$ [ $-0.8$ , $+2.9$ ]	$\mathbf{+3.4}$ [ $+1.3$ , $+5.4$ ]	$+1.2$ [ $-0.9$ , $+3.3$ ]	$-0.8$ [ $-3.3$ , $+1.7$ ]
DeepSeek-R1-32B	$-1.9$ [ $-7.3$ , $+3.4$ ]	$+1.4$ [ $-3.7$ , $+6.6$ ]	$-4.7$ [ $-10.2$ , $+0.9$ ]	$-5.4$ [ $-10.8$ , $+0.1$ ]	$-1.9$ [ $-7.5$ , $+3.7$ ]	$-3.1$ [ $-8.6$ , $+2.3$ ]
MedGemma-27B	$-1.6$ [ $-6.5$ , $+3.3$ ]	$+1.9$ [ $-3.0$ , $+6.8$ ]	$\mathbf{-8.5}$ [ $-13.8$ , $-3.2$ ]	$\mathbf{-5.5}$ [ $-10.8$ , $-0.3$ ]	$-2.9$ [ $-8.0$ , $+2.2$ ]	$\mathbf{-6.2}$ [ $-11.3$ , $-1.2$ ]

Appendix

Table 27: Accuracy gap between originally-female and originally-male patients (F

-

M, percentage points, 95% CI) at baseline and after gender swap. Bold marks intervals excluding zero.

ID	Annot.	Region	Yrs	Specialty
P01	736	US	12	Family Medicine
P02	736	US	9	Internal Medicine
P03	736	US	14	Family Medicine
P04	736	US	11	Family Medicine
P05	736	US	16	Family Medicine
P06	736	US	8	Family Medicine

Appendix

Table 28: Physician annotator panel. All annotators hold an MD degree.

Figure 6: Labeling Instructions Page 1

Figure 7: Labeling Instructions Page 2

Figure 8: Labeling Instructions Page 3

Explore similar work

CARE-Bench: Benchmarking Patient-Facing LLM Triage
Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaTriage Disclosures
1Harvard University · University of Technology Sydney
Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate
May 28, 2026David Fraile Navarro, Berardino Como, Jialei Sheng +2Large Language Models Fail Multiple-Choice Questions
Macquarie University, Sydney, Australia · Politecnico di Bari, Bari, Italy · NSW Health, Sydney, Australia +1
Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support
Jul 29, 2026Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem +10Triage Clinical Decision Support
Atman Labs · Oxford University Hospitals · Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford +11

Aug 4, 2026, cs.AIJ/K move · Enter open · S save

CARE-Bench: Benchmarking Patient-Facing LLM Triage

Yining Hua, Hongbin Na, Cyrus Ayubcha

1Harvard University · University of Technology Sydney

Triage Disclosures Action Space

Open paper AskPDF arXiv

Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.

Abstract

Explore similar work

CARE-Bench: Benchmarking Patient-Facing LLM Triage

Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate

Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support