OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

Anqi Li; Zhixuan Ge; Yixuan Duan; Jiarong Qian; Chi-Yu Chen; MingYu Lu; Huan-Yu Hsu; Yu Gu

cs.CLSep 26, 2026

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

Authors: Anqi Li, Zhixuan Ge, Yixuan Duan, Jiarong Qian, Chi-Yu Chen, MingYu Lu, Huan-Yu Hsu, Yu Gu, +4 more

Organizations: Rice University · University of Illinois at Urbana-Champaign · University of Washington · National Yang Ming Chiao Tung University · Microsoft · University of Pennsylvania

Abstract

Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these real-world trajectories. We introduce OpenTumorBoard, a benchmark with 611 patient cases and 19,157 discussion turns across ten specialist roles, transcribed from 12,534 minutes of publicly available tumor board recordings on YouTube. The benchmark evaluates two settings: SPECIALIST TURN, in which an LLM responds to a clinically significant question posed during a real discussion, and BOARD SIMULATION, in which it generates an entire back-and-forth discussion and reaches a consensus on therapy recommendations, surgical plans, next actions and clinical trial matching. Evaluation of 14 general-purpose frontier and medical LLMs reveals substantial limitations: the best models score 3.43 out of 5 in clinical equivalence to specialist answers and 2.78 out of 5 in alignment with recorded board conclusions. Supervised finetuning and reinforcement learning improve performance on a held-out test set, suggesting that real-world discussion trajectories can support model adaptation. Three M.D. experts review a subset of the benchmark, finding high information coverage and factuality of patient cases and strong fidelity of extracted consensus conclusions. We will release OpenTumorBoard and its automated curation pipeline to support the development and evaluation of LLMs for multidisciplinary, personalized cancer decision-making.

Figures & tables

Figure 1: Overview of OpenTumorBoard. Each LLM simulates a virtual multidisciplinary tumor board that reviews patient cases spanning 11 clinical modalities, engages in back-and-forth discussion, and reaches consensus on cancer care.

Dataset	Real-world tumor board cases	Multi- disciplinary team	Ground- truth discussion	Multimodal data	No. of tumor board cases
MedQA ( Jin et al., 2020 )	✗	✗	✗	✗	N/A
CRAFT-MD ( Johri et al., 2025 )	✗	✗	✗	✓	N/A
MediQ ( Li et al., 2024 )	✗	✗	✗	✗	N/A
AgentClinic ( Schmidgall et al., 2025 )	✗	✗	✗	✓	N/A
MedAgentBench ( Jiang et al., 2025 )	✗	✗	✗	✗	N/A
MedAgentGym ( Xu et al., 2025 )	✗	✗	✗	✗	N/A

Table 1: Biomedical benchmarks for clinical decision-making. Real-world tumor board cases require cases that originate in actual tumor-board practice.

Figure 2: Automated construction of OpenTumorBoard. The pipeline transcribes and diarizes each tumor board recording, segments the discussion into patient cases, and extracts slides from video frames. It then infers the clinical role of each speaker, temporally aligns utterances with the displayed evidence, and removes information leakage to construct multimodal inputs for Specialist Turn and Board Simulation .

Figure 3: A , the discussion trajectory of a representative case across 10 specialist roles. B , Distribution of OpenTumorBoard by cancer site, modality, specialist role and specialist turn type.

Figure 4: Specialist Turn performance by 9 specialist turn types. The heatmap reports mean clinical equivalence to real specialist responses across 4,844 test questions. The left panel reports average response length, with error bars indicating standard deviation.

	Specialist Turn			Board Simulation
	Question relevance	Response correctness	Discussion support	Case coverage	Case factuality	Consensus fidelity
High	76.5%	92.6%	94.4%	100%	100%	100%
Medium	18.3%	4.9%	3.8%	0%	0%	0%
Low	5.1%	2.5%	1.8%	0%	0%	0%
Annotator agreement	60.0%	79.2%	79.4%	80.0%	93.3%	73.3%

Table 2: Expert audit of the ground truth: 447 question-answer pairs, 15 case summaries, and 15 consensus conclusions. Rating distributions reflect final ratings after adjudication. Annotator agreement is the percentage of items rated identically by the two primary raters before adjudication.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Metric	Value	Items
Case segmentation	Case count	100.0%	15 recordings
	Case boundary	98.1%	106 boundaries
Slide extraction	Precision	93.0%	313 extracted frames
	Precision, duplicates excluded	99.7%	313 extracted frames
	Recall	93.6%	311 presented slides
Role inference	Accuracy	96.3%	136 speakers

Appendix

Table A1: Audit of the curation pipeline. Three annotators independently reviewed fifteen randomly selected recordings and every item was resolved by a majority of the three. Precision is over the frames the pipeline extracted, where a duplicate is a near-copy of a frame already extracted. Recall is over the slides the annotators recorded as presented, with a missed slide accepted only on unanimous report or on adjudication. The case boundaries are the start and the end of each of the 53 cases in these recordings.

Model	Input	Equiv.	Crit. err.	Unsupp.	ROUGE-L	BERTScore	Tokens
DeepSeek-V4-Pro (R)	caption	3.43	5.6%	11.8%	0.183	0.616	637
DeepSeek-V4-Flash (R)	caption	3.33	6.7%	13.7%	0.181	0.617	325
Gemma 4 31B (R)	image	3.08	6.5%	10.0%	0.198	0.633	771
Ministral 3 14B	image	3.02	12.1%	28.7%	0.169	0.608	91
Meditron3-70B	caption	3.01	11.2%	20.9%	0.217	0.643	75
Llama 4 Scout	image	2.97	10.2%	15.0%	0.209	0.642	63

Appendix

Table A2: Specialist Turn , every configuration, sorted by clinical equivalence (1–5) over the 4,844 test questions. (R) marks a reasoning run. Input is what the model saw for each slide, the image with its caption or the caption alone. Critical error and unsupported claim are the judge’s two flags as a share of answers. ROUGE-L and BERTScore are against the specialist’s real answer. Tokens is the mean output length per answer.

Model	Input	Align.	Malformed	ROUGE-L	BERTScore	Tokens
Gemini 3.7 Flash (R)	image	2.78	0	0.177	0.641	2,759
Grok 4.6 (R)	image	2.58	0	0.159	0.610	4,932
Qwen3.8-Max (R)	image	2.57	0	0.150	0.603	11,276
GPT-5.6 Sol (R)	image	2.53	3	0.149	0.612	8,213
Claude Opus 5 (R)	image	2.53	0	0.140	0.601	5,540
Gemma 4 31B (R)	image	2.52	0	0.181	0.636	1,693

Appendix

Table A3: Board Simulation , every configuration, proprietary models above the rule and open-weight models below, each block sorted by conclusion alignment (1–5) over the 184 test cases. (R) marks a reasoning run. Malformed is the number of outputs with no conclusion block to extract, which stay in the mean at 1. ROUGE-L and BERTScore are computed on the extracted conclusion against the board’s. Tokens is the mean output length per case, discussion included.

Model	Conclusion alignment	Malformed	Four decisions	Therapy recommendation	Surgical plan	Next action	Clinical trial matching
Qwen2.5-VL-3B	1.58	22	1.78	1.68	1.83	1.88	1.72
+ SFT	1.67	3	2.11	1.78	2.42	2.37	1.89
+ SFT + RL	1.86	4	2.23	1.95	2.69	2.44	1.84

Appendix

Table A4: Training on OpenTumorBoard, the three arms of Section 3.3 on the 184 test cases. Conclusion alignment scores a malformed output at 1. Four decisions is the macro mean of the four-decision rubric over its four decisions, each a mean over the cases where the board took that decision, with a malformed output scored 1 on every decision in scope.

Specialist turn type	$n$	Mean of 9	Best	Crit. err.	Unsupp.
Clarification question	178	3.30	3.65	5.1%	8.4%
Findings interpretation	1,284	3.17	3.65	2.9%	10.3%
Agreement or support	178	3.07	3.42	10.7%	8.4%
Next action suggestion	654	2.97	3.46	2.8%	5.7%
Uncertainty	339	2.95	3.38	2.1%	9.7%
Eligibility assessment	322	2.95	3.48	7.5%	9.9%

Appendix

Table A5: Specialist Turn by specialist turn type. Clinical equivalence (1–5) averaged over the nine configurations and for the best one, DeepSeek-V4-Pro with reasoning, with that configuration’s critical-error and unsupported-claim rates. Treatment recommendation, clinical trial suggestion and evidence discussion score lowest, and the unsupported-claim rate of the best configuration rises from 11.8% overall to 31.5% on clinical trial suggestion and 23.5% on evidence discussion.

Explore similar work

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1Multidisciplinary Tumor Boards Clinical
Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA
TumorBoard: Evidence-Grounded Multi-Agent Decision Support for Longitudinal Neuro-Oncology
Aug 4, 2026Yantong Liu, Zheyu Zhang, Runpeng Liu +3Multidisciplinary Tumor Boards Clinical Decision Support
Department of Computer Information Engineering, Kunsan National University, Gunsan 54150, Republic of Korea · School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 14, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Diagnostic Benchmark
MBZUAI · Cairo University · Ain Shams University +2

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

Negin Ashrafi, Jia Luo, Stacey M. Frumm, Roxana Daneshjou

Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA

Multidisciplinary Tumor Boards Clinical Model Auditing

Open paper AskPDF arXiv

Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology. As AI enters this workflow, a key safety challenge is distinguishing truly unsupported recommendations from evidence-supported options that still require oncologist review because of incomplete information, poor ECOG performance status, or other clinical caveats. We introduce OpenMTB-Audit, an open-source benchmark of 500 synthetic non-small cell lung cancer cases spanning five adversarial error categories and four safety labels: Supported, Partially Supported, Unsupported, and Insufficient Information. Across eight large language model configurations, we identify pervasive over-refusal: all LLM configurations failed to retain the Partially Supported label in 83.3-100% of true Partially Supported cases, achieving high aggregate safety scores through label collapse rather than clinically calibrated reasoning. To address this limitation, we developed MTB-AuditAgent, a deterministic seven-module framework separating evidence verification, missing-information detection, safety classification, and abstention. It reduces over-refusal to 6.7% and achieves 91.2% accuracy (95% CI: 88.6-93.6%). A two-oncologist annotation study found disagreement concentrated at the boundary between information sufficiency and treatment optimization, underscoring the need to preserve clinically meaningful distinctions.

Abstract

Explore similar work

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

TumorBoard: Evidence-Grounded Multi-Agent Decision Support for Longitudinal Neuro-Oncology

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations