NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

Pengkai Wang; Wei-Wei Zhang; Yan Li; Min Tang; Zhitian Hou; Zeyu Liu; Guanghao Zhu; Yuanyi Wang

cs.AINov 5, 2026

NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

Authors: Pengkai Wang, Wei-Wei Zhang, Yan Li, Min Tang, Zhitian Hou, Zeyu Liu, Guanghao Zhu, Yuanyi Wang,

Organizations: The Hong Kong Polytechnic University · Sun Yat-sen University · Sun Yat-sen University Cancer Center · Monash University

Abstract

Current medical benchmarks typically break down clinical competence into isolated factual questions, single-image tasks, or static case vignettes. These tests score individual answers, but reveal less about whether a model can integrate subspecialty knowledge with multimodal evidence and track a patient's condition throughout a course of care. To close this gap, we introduce NPCBench, a clinical apprenticeship benchmark for nasopharyngeal carcinoma (NPC). NPCBench mirrors specialist training, from guideline and foundational knowledge to case planning, longitudinal review, CarePath integration, and open-ended consultation. It contains 4,777 component and consultation items and a 100-patient CarePath cohort with 417 scored diagnostic and care tasks. The episode-level evaluation tests whether diagnostic and care answers are jointly correct within a care pathway. By linking stepwise tasks to patient-level care trajectories, NPCBench evaluates whether models ground decisions in visual evidence, interpret changes in patient state, and meet clinical requirements in consultation. Across 30 evaluated models, we observe a composition gap: strong performance on individual tasks does not ensure jointly correct diagnostic and care answers for the same patient. Among 21 multimodal models, strict CarePath success reaches at most 23%. Paired controls and physician calibration further examine evidence grounding, longitudinal revision, and consultation judging. Overall, NPCBench broadens the evaluation of medical artificial intelligence beyond static question-answering paradigms to an apprenticeship-oriented assessment of clinical workflows, thereby offering a testbed for evidence-based and temporally coherent reasoning in oncology care.

Figures & tables

Figure 1: Overview of NPCBench. (a) NPC care combines pathology, MRI, biomarkers, and clinical records under guideline constraints. (b) The apprenticeship framework spans component skills, longitudinal revision, and CarePath evaluation; O1 assesses consultations. (c) Changing patient states drive reporting of local and joint correctness. CarePath evaluates each decision using the provided clinical context; separate serial controls examine feedback from prior answers.

Figure 2: Task composition of NPCBench. Evaluation items are distributed across six progressive levels (L1–L6) and the open-ended consultation suite (O1). Hatched slices each account for less than 5% of their subset; L6 shows its 217 care decisions. More clinical category names are defined in Appendix A .

Figure 3: Construction of the L1–L5 component tasks. Guideline questions, image tasks, and clinical checkpoints undergo specialist review of the available evidence and reference answers.

Figure 4: Staging, longitudinal decisions, and patient-level correctness. (a) , L3 T/N accuracy on separate 400-patient cohorts. (b) , L5 decision accuracy (Turn) and all-turn success: 305 decisions, 100 patients. (c) , L6 Care accuracy and All-care success: 217 decisions, 100 patients. (d) , Pathology (Path.) and MRI T accuracy; majority-class baselines are 99% and 58%, respectively. (e) , Joint pathology/T correctness (Path.+T), All-care, and their intersection (Strict). All-turn and All-care require every decision correct. Panels (c–e) use the same 100 patients; gray curves show other models.

	Targeted Audit ( $n=548$ )		Expanded Audit ( $n=1{,}013$ )
Judge	Agree.	$\kappa$	Agree.	$\kappa$
GPT-4.1	77.7	0.553	77.6	0.547
GPT-OSS-120B	81.4	0.610	70.5	0.387
Qwen3-30B	73.2	0.472	76.9	0.538
Qwen3.6-27B	79.4	0.578	77.8	0.544

Table 2: Physician–judge agreement. Agreement (%) and Cohen’s

\kappa

are measured against physician annotations. Targeted Audit (

n=548

) and Expanded Audit (

n=1{,}013

) denote the two calibration sets;

n

counts criterion-level labels.

Figure 5: Consultation quality under the GPT-OSS-120B judge. (a) , Major-error rate and C-Rub. score; major errors contribute to score deductions. (b) , Item-mean and case-mean scores for the same 103 consultations from 42 cases. (c) , Clinical tags: adverse events (AE), diagnosis (Dx), elderly (Eld), pediatric (Ped), treatment (Tx), and workup (Wk), with two to 43 questions per tag. (d) , Criterion-level rates: core answers, supporting evidence, and conditional requirements satisfied; major and minor error criteria not triggered. These error-avoidance rates differ from the response-level major-error rate in (a).

Figure 6: Diagnostic and care correctness in 100 patients. (a) , All-care versus Strict for 21 models; diamonds mark the four construction models. (b) , Joint outcomes for four models. Care means all care answers are correct; Image requires correct pathology and T labels; Both equals Strict.

Figure 7: Evidence, decision changes, and evaluation conditions. (a),(f) , Evidence-grounded T success (Grounded T) and fixed-rule staging on 100 patients. (b),(g) , Corrections, regressions, and gains from snapshot to full history. (c),(h) , L5 all-turn and L6 Strict success by decision count (

n

: patients). (d) , Care change with reference pathology, MRI T stage, and invasion findings. (i) , MCQ versus short-answer accuracy on 71 matched questions; gray denotes GPT-4.1. (e),(j) , Judge sensitivity/specificity (Se/Sp) and agreement on requirements (Req.), major errors (Maj.), and minor errors (Min.). Points represent models or judges; OSS-120B is GPT-OSS-120B. Changes are percentage points (pp); bars show 95% patient-cluster (d) or response-cluster (e,j) intervals.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Description
Measurement target	Guideline application, image interpretation, selected care decisions, and consultation quality in NPC; component scores and joint patient correctness are reported separately.
Intended use	Research benchmarking, model comparison, clinical error analysis, and evaluation-method development.
Out-of-scope use	Direct diagnosis, treatment recommendation, triage, or patient-specific clinical decision-making without physician supervision.
Data boundary	Guideline-derived questions, expert-authored vignettes, de-identified patient records, pathology/MRI evidence, longitudinal checkpoints, selected CarePath decisions, and consult rubrics.
Privacy handling	Names, contact details, addresses, and institution-specific identifiers are removed. Dates use relative clinical time where appropriate; L4 retains dates needed to determine the applicable guideline edition. Clinically relevant age, sex, and time intervals are preserved.
Evaluation controls	Shared prompts, adjudicated reference answers, final-answer extraction, O1 judge comparisons, and specialist review of answer clarity.

Appendix

Table A1 : Benchmark scope, data handling, and intended use.

Subset	Evaluation unit	Scale	Source material	Reference annotation
L1	Guideline-memory MCQ	250 questions	NPC guidelines	Expert-validated guideline knowledge
L2	Short-vignette MCQ	185 questions	Clinical vignettes	Specialist-reviewed answers
L3	Pathology/MRI multimodal tasks	69 pathology overview questions; 2,700 evaluated MRI cards; 400 T-stage cases; 400 N-stage cases; 200 MRI sequence questions	Pathology and MRI images	Expert-validated pathology, imaging, and staging labels
L4	Case-level MCQ	165 questions over 84 patients	Clinical vignettes	Specialist-reviewed plans
L5	Sequential patient-turn MCQ	100 real patients; 305 questions	De-identified longitudinal records	Physician-adjudicated decision turns
L6	Selected CarePath decisions	100 patients; 417 scored tasks (100 pathology, 100 MRI T, 217 care)	Clinical episodes	Adjudicated care decisions; 75 pathology proxies

Appendix

Table A2 : Task scale, source materials, and reference annotation.

Episode source	Patients	Care nodes	Pathology evidence	Treatment-selection coverage
Matched pathology	25	115	Patient-matched specimens	Includes selected initial-treatment decisions
Assigned pathology	75	102	Assigned specimen proxies	No scored initial-treatment-selection question
CarePath cohort	100	217	25 matched; 75 proxy	Six treatment items are excluded from scoring

Appendix

Table A3 : Membership and evidence linkage within the CarePath cohort. All 100 patients have matched MRI and clinical records; the image-label comparison uses 146 fixed questions for the 25 patients with matched pathology.

Depth	Patients	Turns	Patients	Care nodes	Node share (%)
	L5		CarePath cohort
1	13	13	52	52	24.0
2	19	38	23	46	21.2
3	31	93	3	9	4.1
4	24	96	7	28	12.9
5	13	65	8	40	18.4

Appendix

Table A4 : Scoring depth and the weight of each patient group. Depth is the number of scored decisions for one patient. Patient-level success weights patients equally; turn/node accuracy weights the scored decisions equally. L5 and the CarePath cohort are separate evaluation cohorts.

Clinical group	Questions	Core	Evidence	Conditional	Major	Minor
Treatment	43	182	171	152	140	148
Adverse-event management	35	127	105	69	104	92
Diagnosis	11	31	51	22	52	37
Special populations	12	53	49	36	36	34
Workup	2	8	10	3	5	9
Total	103	401	386	282	337	320

Appendix

Table A5 : Consultation questions and rubric criteria by clinical group. Positive requirements comprise core answers, supporting evidence, and conditional recommendations. The 1,069 positive and 657 error criteria sum to 1,726.

Evaluation	Scored units	Evidence and history	Endpoint
L3 direct staging	400 T-stage and 400 N-stage patients	Structure cards or same-plane sequence groups; AJCC/UICC edition 8	T and N accuracy; anatomical diagnostics on 100 T cases
CarePath cohort	100 patients; 217 care decisions	AJCC edition 9 MRI; predicted pathology and T-stage labels and supplied clinical context; no prior care answers	Decision-weighted Care; patient-level Strict
Image-evidence control	25 patients; 146 care decisions	Predicted versus reference image findings; other clinical context fixed	Paired change in patient-macro care accuracy
Serial control	25 patients; 146 care decisions	Reference image labels; persistent versus stateless interaction	Paired change in all-care correctness
L5 history control	100 patients; 305 turns	Full history versus a snapshot with historical time markers removed and records regrouped	Turn-accuracy change and all-turn correctness

Appendix

Table A6 : Inputs and endpoints in the linked and matched evaluations. The CarePath cohort contains 417 scored tasks, including 217 care decisions. Image-evidence replacement and serial-interaction analyses evaluate 146 care questions for 25 patients within this cohort.

Model	Input interface	Inference
Medical-Specific LLMs
HuatuoGPT-o1-72B	Text	Local
HuatuoGPT-o1-7B	Text	Local
Baichuan-M2-32B	Text	Local
Fleming-R1-32B	Text	Local
Fleming-R1-7B	Text	Local

Appendix

Table A7 : Evaluated models and task interfaces. Text models cover L1/L2/L4/L5/O1; image-capable models additionally cover L3 and L6.

\dagger

marks the four models used during CarePath cohort construction.

Subset	Input	Response	Scoring unit and endpoint
L1	Guideline MCQ	Option letter	Item accuracy over 250 questions
L2	Compact vignette and MCQ	Option letter(s)	Macro accuracy over four capability groups
L3	Pathology overview; MRI panels or structure cards	Diagnostic labels and structured findings	Pathology: case; T/N stage: patient; sequence: item; structure: card accuracy
L4	Single-case history and MCQ	Option letter(s)	Patient-macro accuracy over 84 patients
L5	Longitudinal history through the current checkpoint	Option letter	Turn accuracy and all-turn patient success; 305 turns, 100 patients
L6	Images, clinical context, and care questions	Diagnostic labels and option letters	Care-node accuracy and strict patient success; 217 nodes, 100 patients

Appendix

Table A8 : Task interfaces and scoring units used in the main NPCBench table. Formal definitions follow in sections C.2.1 and C.5.1 .

Symbol	Meaning and domain
$\mathcal{I},\ n=\|\mathcal{I}\|$	Scored item set and its size; $i\in\mathcal{I}$ indexes one question, image item, or turn.
$y_{i},\hat{y}_{i},\ a_{i}$	Reference label, parsed model label, and correctness $a_{i}=\mathbf{1}\{\hat{y}_{i}=y_{i}\}\in\{0,1\}$ .
$P,\ p$	Number of evaluated patients and patient index $p\in\{1,\ldots,P\}$ .
$\mathcal{S}_{p},\ n_{p},\ N$	Scored L6 care-node set for patient $p$ , its size $n_{p}=\|\mathcal{S}_{p}\|\geq 1$ , and total scored nodes $N=\sum_{p}n_{p}$ .
$c_{pj}$	Correctness of L6 care node $j\in\mathcal{S}_{p}$ for patient $p$ ; a binary indicator.
$d_{p},\ t_{p}$	Binary correctness indicators for pathology and MRI T stage; these symbols are not the predicted class labels.

Appendix

Table A9 : Notation for closed-answer and patient-level evaluation. Each task uses the same finalized cohort and scoring set for every evaluated model.

Figure A1 : Family-level capability profile. Family medians (a) and task-wise best scores within each family (b); the latter may come from different models. MM-EG averages five L3 endpoints; L6 uses the CarePath cohort. Per-model results appear in Table 1 and Fig. A2 .

Model	Turn	Patient macro	All-turn success
API models
DeepSeek-V3.1	74.4	74.2	42.0
GPT-5.1	75.7	73.7	45.0
GPT-4.1	78.7	77.5	45.0
Gemini-3.1-Pro-Preview	87.9	88.6	71.0
Gemini-2.5-Pro	82.0	81.6	56.0

Appendix

Table A10 : Longitudinal decision performance on 100 patients and 305 questions (%). Turn accuracy weights decisions equally; patient macro weights patients equally; all-turn success requires every scored answer for a patient to be correct.

Figure A2 : Model performance by clinical capability. Scores follow Table 1 ; gray cells denote unavailable interfaces. L6 uses the CarePath cohort, and L3 uses its static-image cohorts.

Metric	Scoring target	Interpretation
Direct T	The final t_stage field generated by the evaluated model in the patient-level MRI prompt	Measures whether the model can output the correct stage label, but does not require the supporting structure-level evidence to be correct
Complete Inv.	Whether all 18 base-structure invasion labels are correct for one patient	Measures whether every scored invasion finding is correct
EG T	Complete Inv. plus correct Direct T for the same patient	Strict evidence-grounded staging success; a model must both extract the evidence and produce the right stage
Oracle-slot T	Deterministic staging after placing predicted invasion labels into the gold structure slots	Isolates invasion recognition while removing structure-localization errors
Model-slot T	Deterministic staging after placing predicted invasion labels into the model-predicted structure slots	Measures the full structure-localization-plus-invasion evidence chain
LLM-slot T	Fixed GPT-OSS-120B staging from model-generated structure-card summaries	Tests whether a stronger text stager can recover T stage from the generated evidence representation

Appendix

Table A13 : Anatomical diagnostic panel (100 patients). Interpretation of the L3 MRI staging metrics and ablations. These metrics are reported together because each isolates a different part of the evidence-to-stage chain.

Figure A3 : L3 evidence-grounding diagnostics. Panels (a,b,d) use the 100-case anatomical panel; panel (c) uses the GPT-OSS-120B text-stager experiment. (a) Direct T-stage accuracy can diverge from class-balanced T recall. (b) Base-structure localization does not guarantee correct structure-plus-invasion evidence. (c) When gold 18-structure evidence is supplied, a fixed GPT-OSS-120B stager nearly reaches the rule ceiling; when model-generated evidence is supplied, LLM-slot T-stage scores remain much lower. Reference shows gold-evidence LLM-slot accuracy and balanced recall, with no direct-image prediction. (d) Per-stage recall shows uneven recognition of the four T categories.

Model	Path.	Struct.	Inv.	Joint	Pt. Inv.	EG T	Seq.
HealthGPT-Pro-4B	56.5	9.4	30.4	2.9	43.3	0.0	54.0
HealthGPT-Pro-8B	56.5	8.6	41.6	3.6	48.4	0.0	50.0
Hulu-Med-4B	56.5	5.6	30.9	1.7	47.7	0.0	30.0
Hulu-Med-7B	56.5	7.5	48.0	3.6	62.4	5.0	58.0
Hulu-Med-32B	60.9	21.8	46.6	10.1	59.8	3.0	75.5
Lingshu-7B	56.5	9.3	6.8	0.6	57.9	4.0	47.0

Appendix

Table A14 : Local L3 component results. Structure and joint card accuracies weight all 2,700 cards equally. Inv. is conditioned on correct structure identification. Pt. Inv. averages correctness over the 18 invasion labels per patient. EG T requires all 18 labels and the T category to be correct. Pathology and sequence match Table 1 ; all values are percentages.

Model	Complete Inv.	Direct T	EG T
HealthGPT-Pro-4B	0.0	39.0	0.0
HealthGPT-Pro-8B	0.0	33.0	0.0
Hulu-Med-4B	0.0	35.0	0.0
Hulu-Med-7B	5.0	15.0	5.0
Hulu-Med-32B	3.0	16.0	3.0
Lingshu-7B	4.0	35.0	4.0

Appendix

Table A15 : MRI diagnostics on the anatomical panel for 100 patients. Complete Inv. requires all 18 invasion labels to be correct; Direct T requires the correct final T-category field; EG T requires both. Missing or invalid labels remain incorrect in their respective endpoints.

Model	T1	T2	T3	T4	Direct T	Bal. T	T-bal60
HealthGPT-Pro-4B	6.7	6.7	11.4	94.3	39.0	29.8	31.7
HealthGPT-Pro-8B	0.0	0.0	5.7	88.6	33.0	23.6	25.0
Hulu-Med-4B	26.7	13.3	8.6	74.3	35.0	30.7	26.7
Hulu-Med-7B	100.0	0.0	0.0	0.0	15.0	25.0	25.0
Hulu-Med-32B	80.0	0.0	2.9	8.6	16.0	22.9	20.0
Lingshu-7B	73.3	0.0	0.0	68.6	35.0	35.5	36.7

Appendix

Table A16 : T-category diagnostics on the anatomical panel. The 100 patients comprise 15 T1, 15 T2, 35 T3, and 35 T4 cases. Balanced T averages the four class recalls. T-bal60 uses a fixed subset of 15 cases per stage, with the same patient membership for all models.

Model	Struct.	Inv.	Joint	Gold Rule	Oracle-slot	Model-slot	Direct T
HealthGPT-Pro-4B	9.4	30.4	2.9	99.0	35.0	26.0	39.0
HealthGPT-Pro-8B	8.6	41.6	3.6	99.0	35.0	23.0	33.0
Hulu-Med-4B	5.6	30.9	1.7	99.0	35.0	15.0	35.0
Hulu-Med-7B	7.5	48.0	3.6	99.0	30.0	28.0	15.0
Hulu-Med-32B	21.8	46.6	10.1	99.0	52.0	35.0	16.0
Lingshu-7B	9.3	6.8	0.6	99.0	35.0	33.0	35.0

Appendix

Table A17 : Deterministic evidence-to-stage analysis on the 100-patient anatomical panel. Inv. is conditional on correct anatomy; Joint requires both anatomy and invasion. Oracle-slot places predicted invasion in the reference anatomy slot; Model-slot uses predicted anatomy. Unfilled slots take the fixed zero default, whereas complete-evidence correctness requires every label. Gold Rule measures the ceiling of the 18-structure mapping.

Model	Direct T	Rule-slot T	LLM-slot T	Bal. LLM T
Lingshu-7B	35.0	33.0	32.0	23.8
Lingshu-32B	35.0	35.0	35.0	25.0
InternVL3.5-38B	33.0	15.0	15.0	25.0
Qwen3-VL-8B-Instruct	35.0	35.0	35.0	25.0
Qwen3-VL-30B-A3B-Instruct	35.0	33.0	33.0	23.5
HealthGPT-Pro-4B	39.0	26.0	26.0	25.2

Appendix

Table A18 : LLM-stager ablation for structure-evidence-based MRI T staging. All 21 models contribute evidence summaries for the same 100 patients. A fixed GPT-OSS-120B stager infers T stage from each model’s anatomical labels, invasion predictions, and aggregated votes. Gold 18-structure evidence yields 99.0 LLM-slot T and 99.3 balanced T from these anatomical findings.

Figure A4 : Class balance and scoring depth in the CarePath cohort. T1–T4 counts are 7/12/58/23; pathology labels are 99 positive and one negative. For 1/2/3/4/5/6 scored care decisions, patient counts are 52/23/3/7/8/7. These distributions belong to the selected L6 cohort; L3 has different populations and a different MRI staging edition.

Figure A5 : Local and joint correctness in the CarePath cohort. (a) Care accuracy uses 217 decisions; strict success uses 100 patients. (b) Nested intersections count the same patients: correct pathology, correct pathology and T stage, then all selected care answers as well. Each successive requirement narrows the set of successful patients. (c) Exact binomial 95% intervals for the four representative models. Panels (a,b) include all 17 models not used in selection; the remaining 13 are gray. All 21 model scores appear in Table A19 .

Model	$\|\mathcal{C}\|$	$\|\mathcal{F}\|$	$\|\mathcal{C}\cap\mathcal{F}\|$	$\|\mathcal{C}\cap\mathcal{F}^{c}\|$	$\|\mathcal{C}^{c}\cap\mathcal{F}\|$	$\|\mathcal{C}^{c}\cap\mathcal{F}^{c}\|$
HealthGPT-4B	8	9	0	8	9	83
HealthGPT-8B †	5	4	0	5	4	91
Hulu-Med-4B	15	8	0	15	8	77
Hulu-Med-7B	19	1	0	19	1	80
Hulu-Med-32B	19	48	10	9	38	43
Lingshu-7B	16	45	9	7	36	48

Appendix

Table A20: CarePath cohort patient-level joint states for all 21 models.

\mathcal{C}

: patients with all selected care answers correct;

\mathcal{F}

: patients with both image labels correct. Entries are set cardinalities; complements are within the same 100 patients. The four intersection counts sum to 100.

\dagger

denotes construction models.

	Matched specimens ( $n=25$ )				Assigned specimens ( $n=75$ )
Model	Care	All-care	Image	Strict	Care	All-care	Image	Strict
Claude-4.6	84.3	52.0	40.0	12.0	76.5	68.0	37.3	26.7
GPT-4.1	75.7	32.0	4.0	0.0	46.1	28.0	8.0	2.7
GPT-5.1	77.4	32.0	16.0	4.0	63.7	50.7	6.7	1.3
Gemini-2.5-Flash	60.0	8.0	28.0	0.0	70.6	60.0	17.3	8.0
Gemini-2.5-Pro	73.9	28.0	28.0	8.0	88.2	85.3	21.3	17.3

Appendix

Table A21 : CarePath cohort outcomes by pathology source. Matched specimens: 25 patients and 115 care decisions; assigned specimens: 75 patients and 102 decisions. Within each group, columns give care accuracy, all-care correctness, both image labels correct, and strict success (%).

\dagger

marks construction models.

Control	Held fixed	Changed	Reported endpoint
L6 image evidence (25 patients)	Patients, care decisions, supplied clinical context	Pathology, MRI T stage, and invaded structures	Patient-macro care difference
L5 history (100 patients)	Baseline/current evidence, question, options, reference answer	Historical time markers and record ordering	Turn-accuracy difference
L1 options (71 pairs)	Physician-audited target knowledge	MCQ versus option-free answer	Paired item-accuracy difference
MRI interface (25 patients)	Model, patients, matched image frames	Static panel versus cine presentation	T and structure differences
O1 Expanded Audit (1,013 labels)	Responses, criteria, physician labels	Automated judge	Agreement and polarity-specific agreement

Appendix

Table A22 : Design of diagnostic comparisons. The O1 comparison applies different judges to the same Expanded Audit response set.

Row	Model	Predicted	Reference	Difference [95% CI]	B/W/S
1	HealthGPT-8B	21.1	17.9	-3.2 [-10.5, 3.8]	8/10/7
2	Lingshu-7B	54.4	53.0	-1.4 [-5.0, 1.8]	2/3/20
3	Hulu-Med-4B	48.0	46.8	-1.3 [-3.9, 1.3]	1/3/21
4	HealthGPT-4B	23.9	22.9	-1.0 [-9.2, 8.0]	7/9/9
5	Qwen3.6-27B	85.5	84.6	-0.9 [-4.4, 2.4]	2/3/20
6	Qwen3.6-35B-A3B	84.2	84.5	+0.2 [-2.5, 2.9]	2/2/21

Appendix

Table A23 : L6 image-evidence replacement: scores and patient changes. Predicted and reference pathology, MRI T stage, and invasion findings are compared on the same 25 selected patients and 146 care decisions. Scores are patient-macro care accuracy (%); differences and paired patient-bootstrap 95% intervals are in pp. B/W/S counts patients with better/worse/unchanged care accuracy under reference inputs. Figure 7 d plots baseline accuracy against the paired difference.

Model	History	Snapshot	Difference [95% CI]	$+/-$
DeepSeek-V3.1	71.5	66.6	+4.9 [1.4, 8.5]	22/7
Fleming-R1-7B	59.7	61.3	-1.6 [-6.5, 3.2]	21/26
GPT-4.1	75.1	75.7	-0.7 [-4.1, 2.9]	14/16
GPT-5.1	71.5	70.5	+1.0 [-3.2, 5.2]	22/19
GPT-OSS-120B	48.9	45.6	+3.3 [-1.6, 8.1]	29/19
GPT-OSS-20B	52.8	48.2	+4.6 [-1.4, 10.7]	46/32

Appendix

Table A25 : L5 full history versus the time-stripped snapshot. The snapshot removes historical time markers and regroups records (305 turns, 100 patients). Accuracy is in percent and differences in pp, with paired patient-bootstrap 95% intervals.

+/-

count turns corrected/harmed by full history.

Figure A6 : Paired controls for evidence and history. (a) Gold versus predicted pathology/MRI inputs: difference in patient-macro care accuracy across 14 local models on the 25 selected patients/146 care decisions. (b) Full history versus the time-stripped, regrouped snapshot: difference in turn accuracy across 20 models on 100 patients/305 turns. Dots show paired differences and bars patient-cluster 95% bootstrap intervals; all models in each control are retained.

Condition	Cisplatin record	Docetaxel record	Timing and position
Full history	Admission 2, day 26; induction, cycle 2; cisplatin 53.0 mg	Admission 2, day 26; induction, cycle 2; docetaxel 105.0 mg	Prefixes retained; records 23 and 24
Snapshot	Induction, cycle 2; cisplatin 53.0 mg	Induction, cycle 2; docetaxel 105.0 mg	Prefixes removed; records 33 and 42
Shuffled order	Admission 2, day 26; induction, cycle 2; cisplatin 53.0 mg	Admission 2, day 26; induction, cycle 2; docetaxel 105.0 mg	Full lines retained; records 19 and 29

Appendix

Table A27 : What the L5 input conditions preserve and change. Excerpts from one actual prompt triplet; English translations condense repeated source-field names. Record positions refer to the 46 historical lines, excluding baseline/current findings. Complete original-language prompts and the line correspondence are included in the accompanying example.

Model	Score	Major	Minor	AE	Dx	Eld.	Ped.	Tx	Workup
GPT-5.1	58.4	14.6	40.8	69.2	42.6	85.4	57.4	51.2	60.1
Gemini-3.1	47.8	12.6	59.2	58.4	50.1	60.5	35.5	39.1	60.1
Claude-4.6	47.5	15.5	58.3	52.1	41.5	76.7	49.8	41.5	63.3
Qwen3.6-27B	47.2	24.3	52.4	61.2	26.5	67.5	39.9	39.6	65.1
Qwen3.5-27B	46.8	19.4	50.5	59.5	31.7	61.3	27.4	42.2	56.7
GPT-OSS-120B	44.9	20.4	49.5	61.8	29.5	51.7	29.3	36.6	59.2

Appendix

Table A29 : Consultation scores and error rates under the GPT-OSS-120B judge. Every model uses 103 questions and 1,726 criteria. Major/Minor are percentages of answers with at least one corresponding error. Clinical-group scores cover adverse events (AE; 35 questions), diagnosis (Dx; 11), elderly patients (Eld.; 4), pediatric patients (Ped.; 8), treatment (Tx; 43), and workup (2). All scores lie in 0–100.

Response model	Sampled resp.	Excluded resp.	Flagged labels retained	Included labels
GPT-5.1	15	0	184	264
Qwen3.6-27B-Instruct	8	0	0	136
GPT-OSS-120B	7	0	102	132
Gemini-3.1-Pro-Preview	18	17	0	16

Appendix

Table A30 : Coverage of the Targeted Audit for O1. Flagged labels retained are included in the final column and are removed only in the 262-label sensitivity analysis.

Judge model	$n$	Acc.	Bal. acc.	$\kappa$	Sens.	Spec.	True rate
GPT-4.1	262	0.767	0.772	0.533	0.802	0.742	0.489
GPT-OSS-120B	262	0.840	0.826	0.665	0.739	0.914	0.363
Qwen3-30B-A3B-Instruct	262	0.752	0.760	0.506	0.811	0.709	0.511
Qwen3.6-27B	262	0.782	0.781	0.558	0.775	0.788	0.450

Appendix

Table A31 : Targeted Audit sensitivity after excluding all flagged responses. Agreement statistics are proportions;

\kappa

is dimensionless. Acc. is overall agreement, Bal. acc. averages sensitivity and specificity, and True rate is the fraction of labels marked true by the judge.

Figure A7 : Physician–judge agreement in the Expanded Audit. All panels use the same 1,013 criterion labels from 54 consultation responses to six cases. (a) Overall agreement and 95% response-cluster bootstrap intervals. (b) Agreement conditional on physician-positive and physician-negative labels (sensitivity and specificity); connecting lines pair the two measures for each judge.

Clinical Reasoning

Clinical reasoning studies how AI systems interpret patient evidence, form and revise diagnoses, and support safe decisions across care settings.

See all 186 papers

Oct 26, 2026

Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

Large language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway. Here we introduce NPC-RubricBench, a phase-aware benchmark for evaluating multistage clinical reasoning across the cancer care continuum, using nasopharyngeal carcinoma as a defined disease context. Curated from longitudinal records of 405 patients across two cancer centers, the benchmark comprises 941 patient-phase units across six sequential care phases and 17,688 patient-specific atomic criteria spanning five evaluation dimensions, with each assessment strictly restricted to decision-time clinical evidence. Evaluating 30 general-purpose and medical LLMs revealed that strong diagnostic performance does not carry over to downstream care. We evaluated 30 general-purpose and medical LLMs, showing that high diagnostic performance did not translate into improved downstream care. Instead, the dominant failure modes reversed across phases. Explicit decision errors dominated anatomical staging at 48.2%, whereas decision omissions dominated initial work-up and post-treatment follow-up at 33.4% and 30.4%, respectively. In the evaluated model set, medically adapted models showed higher omission rates than general-purpose models. Moreover, varying query granularity exposes a critical specification trade-off: broad open-ended requests degrade Decision Quality across all 30 models, while more detailed, checklist-like prompts raise Decision Quality but reduce Evidence Faithfulness and Safety Adherence. Furthermore, NPC-RubricBench demonstrates that static diagnostic benchmarks are insufficient to ensure consistent clinical reliability over time, highlighting the importance of phase-adaptive safeguards in multistage medical AI.

Wei-Wei Zhang, Pengkai Wang, Yan Li +13

Department of Radiation Oncology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, China. +2

Clinical Language Model Evaluation Medical LLMs Clinical Decision-Making Clinical Reasoning Oncology

Ask PDF

Aug 5, 2026

RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care

Background: Respiratory specialty care requires multimodal interpretation, longitudinal risk assessment, guideline-concordant intervention, and whole-course management, which are poorly represented by examination-oriented medical benchmarks. Objective: To develop RESPClinBench, a real-world scenario-based benchmark for respiratory clinical decision-making, and evaluate seven contemporary large language models across AECOPD-PIM and PNBIM. Methods: RESPClinBench cases were adapted from de-identified respiratory clinical data. Three attending-level respiratory physicians revised cases, reference answers, and atomic clinical-action points, while one senior respiratory specialist performed cross-review and final adjudication. AECOPD-PIM comprised 427 open-ended COPD cases, and PNBIM comprised 196 multimodal pulmonary nodule cases combining chest CT with structured clinical information. Seven models generated 4,361 responses through standardized API inference with temperature 0 and a maximum output length of 8192 tokens. An automated framework calculated the final score as the arithmetic mean of atomic-action recall and rubric-based LLM-as-a-Judge assessment. Results: Across 623 cases, the mean final score was 68.58. Qwen3.6-27B ranked first overall at 71.22, Qwen3.5-397B-A17B led PNBIM at 72.48, and Qwen3.6-27B led AECOPD-PIM at 71.11. Imaging hallucination and serious medical risk occurred in 31.85% and 8.16% of PNBIM responses; medication-safety risk and serious medical risk occurred in 26.93% and 1.44% of AECOPD-PIM responses. Conclusions: RESPClinBench identifies task-specific limitations in multimodal pulmonary nodule assessment and longitudinal COPD management. Combining explicit clinical-action coverage, holistic evaluation, and independent safety flags provides a clinically grounded basis for model selection and prospective validation.

Mouxiao Bian, Zhi Chen, Ruiyao Chen +8

Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1

LLM Safety Benchmarks Healthcare Clinical Decision-Making Clinical Language Model Evaluation Multimodal Model Evaluation

Ask PDF

Jul 1, 2026

Evaluating Agentic Harness Systems for Autonomous Computational Pathology

Autonomous computational pathology (ACP) converts high-level pathology analysis goals into executable, traceable and clinically bounded workflows. Realizing this capability requires adapting general agentic harness systems to pathology-specific tasks, tools, evidence standards and clinical claim boundaries. We contribute ACP-Bench, a framework that adapts existing harness systems from computational pathology support toward ACP workflow capability. ACP-Bench evaluates 41 pathology workflow tasks, including 24 biomarker, 7 morphology and 10 prognosis tasks spanning 6 body-system groups and 9 endpoint families. The benchmark evaluates 9 models and 3 harness groups (Claude Code, Codex and Open Code), yielding 369 complete trajectories. ACP-Bench evaluates each trajectory across workflow execution, diagnostic performance and clinical-boundary alignment, combining expert-adjudicated process audits, diagnostic assessment and pathologist-validated safety review. Across evaluated systems, workflow initiation, task interpretation and diagnostic reporting were more mature than tool-bound execution, result binding and reflective workflow revision, and formal end-to-end completion remained rare. ACP-Bench provides a reusable standard for auditing whether agentic systems can operationalize pathology workflows before claims of reliable clinical autonomy.

Jie Lin, Zongyi Chen, Qiaoling Zheng +6

Department of Computer Science at School of Informatics, Xiamen University, Xiamen, China. · National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China. · Department of Pathology, Fujian Medical University Union Hospital, Fuzhou, China. +1

Computational Pathology LLM Agent Evaluation AI Agent Benchmarks AI Agent Safety Benchmarks LLM Agent Harnesses

Ask PDF

Abstract

Clinical Reasoning

Explore similar work

Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care

Evaluating Agentic Harness Systems for Autonomous Computational Pathology