MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis

Feng Guo; Jiaxiang Liu; Yang Li; Qianqian Shi; Mingkun Xu

cs.CVFeb 26, 2026

MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis

Authors: Feng Guo, Jiaxiang Liu, Yang Li, Qianqian Shi, Mingkun Xu

Organizations: Guangdong Institute of Intelligence Science and Technology, Hengqin, China · Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University, Beijing, China

Abstract

Accurate brain tumor diagnosis requires models to not only detect lesions but also generate clinically interpretable reasoning grounded in imaging manifestations, yet existing public datasets remain limited in annotation richness and diagnostic semantics. To bridge this gap, we introduce MM-NeuroOnco, a large-scale multimodal benchmark and instruction-tuning dataset for brain tumor MRI understanding, consisting of 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched multimodal instructions spanning diverse tumor subtypes and imaging modalities. To mitigate the scarcity and high cost of diagnostic semantic annotations, we develop a multi-model collaborative pipeline for automated medical information completion and quality control, enabling the generation of diagnosis-related semantics beyond mask-only annotations. Building upon this dataset, we further construct MM-NeuroOnco-Bench, a manually annotated evaluation benchmark with a rejection-aware setting to reduce biases inherent in closed-ended question formats. Evaluation across ten representative models shows that even the strongest baseline, Gemini 3 Flash, achieves only 41.88% accuracy on diagnosis-related questions, highlighting the substantial challenges of multimodal brain tumor diagnostic understanding. Leveraging MM-NeuroOnco, we further propose NeuroOnco-GPT, which achieves a 27% absolute accuracy improvement on diagnostic questions following fine-tuning. This result demonstrates the effectiveness of our dataset and benchmark in advancing clinically grounded multimodal diagnostic reasoning. Code and dataset are publicly available at: https://github.com/gfnnnb/MM-NeuroOnco

Figures & tables

Figure 1. Overview of MM-NeuroOnco. The benchmark curates 24,726 slices from 20 diverse datasets. Standard diagnostic labels are augmented with fine-grained semantic attributes to construct explicit Chain-of-Thought (CoT) reasoning. This structure enables a multi-grained evaluation paradigm, covering both holistic open-ended diagnosis and targeted closed-ended Question Answering across multiple clinical categories. Overview of MM-NeuroOnco, showing dataset curation across 20 sources, semantic attribute augmentation, and CoT-style reasoning for multi-grained evaluation (open-ended diagnosis and closed-ended QA across clinical categories).

Figure 2. The MM-NeuroOnco dataset curation pipeline, which consists of four sequential steps transforming raw MRI data into semantically enriched multimodal instructions. The MM-NeuroOnco dataset curation pipeline with four sequential steps that transform raw MRI data into semantically enriched multimodal instructions.

Figure 3. Radar chart comparing the performance of the base model and different fine-tuning strategies on MM-NeuroOnco-Bench. Performance comparison on both closed-ended and open-ended QA across multiple LVLMs.

Figure 4. Statistical distributions of the MM-NeuroOnco dataset. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 5. Modality distribution of the MM-NeuroOnco dataset and benchmark.The figure compares the proportional distributions of different MRI modalities in the dataset and the benchmark. An illustration for Figure 4.

General-Purpose LVLMs
Model	Diagnosis	Size	Shape	Spread	Location	Overall
GPT-5.1	37.0	32.3	21.2	58.0	38.1	37.2
Gemini-3-Flash	41.9	29.8	30.8	57.9	41.8	40.9
Claude-Sonnet-4.0	33.2	34.5	28.3	54.4	35.5	35.9
Qwen3-VL-8B	24.3	31.6	30.0	47.2	30.6	30.1
Medical Specialist LVLMs

Table 1. Overall performance comparison on the closed-ended benchmark. Best results are highlighted in bold , and second-best results are underlined . Colored deltas indicate performance changes introduced by Chain-of-Thought (CoT) supervision. Qwen3-VL-8B is evaluated in a zero-shot setting as an open-source baseline.

Model	4-N	5-N	5-R
HuLuMed-32B	59.56	56.90 (−2.66)	49.40 (−10.16)
Lingshu-32B	57.21	55.77 (−1.44)	46.93 (−10.28)
GPT-5.1	59.03	56.74 (−2.29)	49.62 (−9.41)
Average	58.60	56.47 (−2.13)	48.65 (−9.95)

Table 2. Ablation study on the rejection mechanism. N denotes the standard option setting, while R denotes the setting with an explicit rejection mechanism. Colored deltas indicate performance changes relative to the 4-N setting.

Figure 6. Word cloud visualizations illustrating the linguistic distributions of (a) closed-ended questions, (b) open-ended questions, and (c) the evaluation benchmark. Overall pipeline and system overview of the proposed framework.

Figure 7. Heatmap comparing the performance of multiple LVLMs on closed-ended QA tasks across different dimensions. Performance comparison on both closed-ended and open-ended QA across multiple LVLMs.

Figure 8. Three representative case studies for closed-ended and open-ended QA. Supplementary visualization for the data construction pipeline.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9. Distribution of tumor types and modalities.The outer ring shows the distribution of brain tumor types, while the inner ring displays the distribution of MRI modalities. Supplementary Figure 9.

ID	Dataset Name	Samples	Key Characteristics	Link
01	Brain Tumor Classification (MRI)	3,264	4-class MRI tumor classification (glioma, meningioma, pituitary, no tumor).	URL
02	Br35H: Brain Tumor Detection 2020	1,500	Binary tumor classification; balanced dataset.	URL
03	Brain Tumor MRI Dataset	7,023	4-class tumor classification; MRI modality unspecified.	URL
04	Brain Cancer MRI Dataset	6,056	Glioma, meningioma, pituitary tumor cases.	URL
05	BRISC: Brain Tumor Seg. Dataset	6,000	T1-weighted MRI; 4-class labels; segmentation masks.	URL
06	Jun Cheng Brain Tumor Dataset	3,064	T1-weighted contrast-enhanced MRI; three tumor subtypes.	URL

Appendix

Table 3. Inventory of the 20 original data sources incorporated in MM-NeuroOnco.

Figure 10. Coarse-grained text description template converted from structured JSON fields into natural language. Supplementary Figure 10.

Figure 11. Multi-model Assisted Automated Medical Information Extraction and Quality Control Process. Supplementary visualization for multi-source data aggregation and standardization.

Dataset	Sub-Dataset	Description	Size
MM-NeuroOnco	Image Pool	Curated collection of 2D MRI slices from 20 public datasets processed via unified indexing, deduplication, and normalization.	73,226
	Attributes	Integrates 9 LLM-extracted silver labels (e.g., edema and enhancement) alongside human-annotated gold labels such as tumor type.	24,726
	VQA Pairs	Featuring 130k+ closed-ended QA pairs with challenging distractors and CoT reasoning, supplemented by 70k+ open-ended pairs.	200k+
	Benchmark	A high-quality evaluation subset containing 1,000 images annotated with 2k+ closed-ended and 1k+ open-ended questions.	3k+

Appendix

Table 4. Brief description of the four sub-datasets in MM-NeuroOnco and their corresponding data sizes.

General-Purpose LVLMs
Model	Detail	Location	Reasoning	Overall
GPT-5.1	71.15	63.99	86.17	72.72
Gemini 3 Flash	65.71	49.22	82.02	65.67
Qwen3-VL-8B	58.94	59.59	44.28	56.14
Medical-Specialized LVLMs
HuLuMed-32B	60.80	46.58	78.23	61.44

Appendix

Table 5. Overall performance comparison on the open-ended benchmark. Best results are highlighted in bold , and second-best results are underlined .

Figure 12. Data Distribution and Information Extraction Rate. Supplementary visualization for multi-source data aggregation and standardization.

Metric	Value
Reviewed Attribute Fields	136
Attribute-level Precision	89.70%
Information Omission Rate	17.65%

Appendix

Table 6. Silver Annotation Quality Audit Results.

Figure 13. Sample Images of Meningioma, Glioma, Pituitary Tumors, and Others.

Figure 14. Prompt for the Medical Information Extraction Process

Figure 15. Continuation of the Prompt for the Medical Information Extraction Process.

Figure 16. Prompt for Visual Final Review and Quality Control.

Figure 17. Prompt for Automatic Generation of Questions and Options Based on LLM.

Figure 18. Prompt for LLM-assisted Open Question Scoring.

Panel of Figure 19 — Figure 19. Sample Examples of Closed-ended and Open-ended Question Answering from the Instruction Dataset.

Explore similar work

UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
May 16, 2026Shiv Ghosh, Junayd Lateef, Chih-Hua Liu +3VLM Evaluation Medical VQA
Fung Institute for Engineering Leadership University of California, Berkeley Berkeley, CA 94709 · Department of Radiology University of California, San Francisco San Francisco, CA 94158 · Department of Radiology University of California, San Francisco San Francisco, CA 94143 +1
NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
Mar 25, 2026Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska +5Medical VLMs Multimodal Model Evaluation
Faculty of computer science and engineering, Ss. Cyril and Methodius University, Skopje, North Macedonia.
CoRe-BT: A Multimodal Radiology-Pathology-Text Benchmark for Robust Brain Tumor Typing
Mar 4, 2026Juampablo E. Heras Rivera, Daniel K. Low, Xavier Xiong +5Brain Tumor Classification Medical Image Benchmarks
University of Washington, Seattle, Washington 98105, USA · University of Washington School of Medicine, Seattle, Washington 98105, USA · Microsoft Health AI, Redmond, Washington 98052-6399, USA

May 16, 2026J/K move · Enter open · S save

UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation

Shiv Ghosh, Junayd Lateef, Chih-Hua Liu, Yannan Yu+2

Fung Institute for Engineering Leadership University of California, Berkeley Berkeley, CA 94709 · Department of Radiology University of California, San Francisco San Francisco, CA 94158 · Department of Radiology University of California, San Francisco San Francisco, CA 94143+1

VLM Evaluation Medical VQA Oncology Medical Image Benchmarks

Open paper AskPDF arXiv

Brain tumor diagnosis is largely dependent on Magnetic Resonance Imaging (MRI) evaluation, which requires radiologists to synthesize thousands of images across multiple 3D sequences and longitudinal studies. This process requires advanced neuro-radiology training, poses substantial cognitive load, and is highly time-consuming. Despite increasing demands in radiology, this expertise is difficult to scale, straining the current health systems. Vision-Language Models (VLMs) provide an opportunity to reduce this burden through a semi-automated, interactive interpretation of complex brain MRIs. However, they are currently underutilized in neuro-oncology due to a lack of specialized benchmarks for evaluating them. We introduce a clinically relevant visual question answering (VQA) benchmark -- the UCSF-PDGM-VQA dataset -- consisting of 2,387 QA pairs from 473 glioma-related MRI studies in the public UCSF-PDGM dataset. We further establish a performance baseline for six state-of-the-art vision-language models (VLMs) and one large language model on this dataset. We find that current models are incapable of effectively processing multi-sequence, 3-dimensional MRI scans, thus resulting in a suppression of visual features and over-reliance on language priors, causing modality collapse. These findings underscore a critical deficiency in current model reliability and safety within clinical settings, necessitating the development of robust, domain-specific VLMs.

Abstract

Explore similar work

UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation

NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

CoRe-BT: A Multimodal Radiology-Pathology-Text Benchmark for Robust Brain Tumor Typing