SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering

Yaojun Hu; Danyang Tu; Yang Liu; Jiajin Zhang; Wei Fang; Zhiqiang Liu; Chunlai Dong; Yingda Xia

cs.CVAug 26, 2026

SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering

Authors: Yaojun Hu, Danyang Tu, Yang Liu, Jiajin Zhang, Wei Fang, Zhiqiang Liu, Chunlai Dong, Yingda Xia,

Organizations: DAMO Academy, Alibaba Group · College of Computer Science and Technology, Zhejiang University · State Key Laboratory of Transvascular Implantation Devices and TIDRI, Zhejiang University · The Second Affiliated Hospital and Liangzhu Laboratory, Zhejiang University School of Medicine ·

Abstract

Volumetric medical VQA requires reasoning over long and redundant 3D visual token sequences, especially in multi-sequence MRI where complementary modalities provide diverse diagnostic cues but expose the decoder to many repeated anatomical regions. To investigate reasoning under multi-sequence visual redundancy, we first introduce BreMRIs-VQA, a clinically curated breast MRI benchmark with 1.19M QA pairs from 71.0K sequences and 12.9K patients, covering both free-text and multiple-choice questions. We further propose SeVeR, a selective visual exposure framework that compresses dense volumes into modality-wise prototypes and retrieves complementary multi-level evidence with change-aware gated attention during decoding, trained with a marginal-utility self-consistency objective that suppresses unhelpful retrieval. Experiments on BreMRIs-VQA and public benchmarks show that SeVeR improves both discriminative and generative performance while exposing substantially fewer visual tokens.

Figures & tables

Figure 1: Overview of the BreMRIs-VQA benchmark and the proposed SeVeR. (a–b) BreMRIs-VQA contains 6 MRI modalities and 7 categories of clinical tasks derived from 1.19M QA pairs. (c) Architecture of SeVeR, which integrates Greedy Prototype Selection, Change-aware Gated Attention, and Self-Consistency Regularization. (d–e) SeVeR’s superior performance in multiple-choice and free-text generation tasks.

Figure 2: Dataset construction pipeline. The top-right pie chart depicts hierarchical clinical attribute distribution.

Figure 3: Overview of SeVeR. Multi-modal 3D volumes are first encoded into dense tokens via a shared 3D ViT. The Greedy Prototype Selector (GPS) distills redundant tokens into compact prototypes using global affinity to preserve broad visual coverage. Change-aware Gated Attention (CaGA) dynamically injects multi-level features from a bank during decoding, using language-conditioned decoder states to retrieve complementary evidence.

Task	Metric	Zero-shot				Fine-tune
		Qwen3-VL	Hulu-Med	Hulu-Med	Lingshu	M3D-Phi3	M3D-L2	M3D-L3	Lingshu	Qwen3-VL	SeVeR
		4B	4B	7B	32B	4B	7B	8B	7B	4B	4B
Global Background Assessment	Accuracy	23.20	57.25	59.95	69.43	76.78	72.83	66.53	88.23	89.43	89.82
	BLEU	72.76	80.34	82.91	85.12	91.23	92.87	90.12	95.92	96.55	96.63
	ROUGE	72.63	84.76	87.42	91.07	93.45	94.81	92.15	97.42	97.86	97.85
	BERT.	94.41	97.83	98.36	99.27	98.94	99.22	98.83	99.62	99.63	99.68

Table 1: Main performance on BreMRIs-VQA. Bold and underline indicate the best and second-best performance.

Method	Vis. Tokens	Lat. (ms) $\downarrow$	Accuracy	BERT.
Baseline	12544	952.6	69.18	98.42
VisionZip	256	630.8	67.63	98.16
VisionZip	512	698.4	68.72	98.34
VisionZip	1024	829.6	69.03	98.39
DivPrune	256	605.7	67.21	98.08
DivPrune	512	625.6	68.24	98.25

Table 2: Latency–performance comparison with visual token pruning methods on BreMRIs-VQA.

Figure 4: Missing-modality robustness. Multiple-choice accuracy under six modality settings (S1–S6).

Panel of Figure 5 — Figure 5: Interpretability visualization of SeVeR on the question “What is the delayed phase enhancement kinetics of the mass in the upper inner quadrant of the right breast?” with six MRI modalities.

Task	Metric	Zero-shot							Finetune
		Q2.5VL	Q3VL	RadFM	M3D-L2	M3D-P3	OmniV	Lingshu	M3D-L2	M3D-P3	Qwen3-VL	SeVeR
		4B	4B	13B	7B	4B	1.5B	7B	7B	4B	4B	4B
Exist. Det.	Accuracy	19.52	3.52	29.20	18.00	40.25	28.66	59.60	81.09	82.43	82.52	82.62
Static T. Diag.	Accuracy	0.00	0.00	44.11	25.47	25.40	22.96	6.02	51.20	49.30	47.70	51.77
Longit. T. Diag.	Accuracy	0.29	0.15	42.99	24.17	24.31	24.23	12.13	74.78	74.77	74.39	75.28
Medical Measurement	BLEU	1.78	0.11	3.34	15.95	2.55	2.52	2.50	30.54	33.52	35.01	36.78

Table 3: Comparison of zero-shot and finetuned performance across six tasks in 3D-RAD. Exist. Det.: Existence Detection; Static T. Diag.: Static Temporal Diagnosis; Longit. T. Diag.: Longitudinal Temporal Diagnosis. Q2.5VL: Qwen2.5-VL; Q3VL: Qwen3-VL. Bold and underline indicate the best and second-best performance.

Type	Metric	Merlin	M3D-L2	M3D-P3	CT-CHAT	RadFM	SeVeR
Meas	MC	0.256	0.733	0.740	0.740	0.739	0.735
Meas	FT	0.364	0.370	0.366	0.379	0.434	0.521
Recog	MC	0.661	0.664	0.661	0.664	0.789	0.707
Recog	FT	0.664	0.665	0.660	0.663	0.812	0.715
VisRsn	MC	0.382	0.616	0.627	0.620	0.625	0.673
VisRsn	FT	0.392	0.408	0.420	0.428	0.439	0.499

Table 4: Average DeepTumorVQA performance. Meas: Measurement; Recog: Recognition; VisRsn: Visual Reasoning; MedRsn: Medical Reasoning. MC: multiple-choice accuracy; FT: average free-text score.

Setting	Acc.	BLEU	ROUGE	BERT.	$k$
Baseline Ft.	69.45	89.87	89.56	98.58	4608
w/o CaGA	64.36	88.92	85.30	95.42	512
w/o Regul.	68.17	87.04	89.16	97.63	512
Q-GPS	69.92	90.21	89.83	97.66	512
SeVeR	70.57	91.08	90.15	98.70	512

Table 5: Compact ablation summary on BreMRIs-VQA. We report average metrics across seven tasks and the number of visual prototypes exposed to the decoder.

Figure 6: Prototype-budget latency profile. Latency and performance under prototype budgets

k

, with the full-token no-limit setting shown as a dashed reference.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Level–task–attribute flow of BreMRIs-VQA. Ribbon width is proportional to the number of generated QA templates, and colors correspond to the seven clinical workflow task groups.

Figure 8: Question and answer text word clouds for BreMRIs-VQA. Font size indicates token frequency after removing common function words. Unlike the attribute-frequency statistics, this visualization is computed directly from the natural-language question and answer text.

Category	Question Template	Options
Fibroglandular Tissue (FGT)	What is the fibroglandular tissue (FGT) category of the left breast?	A, B, C, D, None
Fibroglandular Tissue (FGT)	What is the fibroglandular tissue (FGT) category of the right breast?	A, B, C, D, None
Background Parenchymal Enhancement	What is the background parenchymal enhancement (BPE) level of the left breast?	Minimal, Mild, Moderate, Marked
Background Parenchymal Enhancement	What is the background parenchymal enhancement (BPE) level of the right breast?	Minimal, Mild, Moderate, Marked
Breast Symmetry	Are the breasts symmetric in size and shape?	Yes, No
Post-surgical Changes	Are there evidence of scar tissue in the left breast?	Yes, No

Appendix

Table 6: Breast-level question templates used for MRI VQA generation.

Category	Question Template	Options
Mass Presence & Location	Is there a mass in the upper inner quadrant of the left breast?	Yes, No
	Is there a mass in the upper outer quadrant of the left breast?	Yes, No
	Is there a mass in the lower inner quadrant of the right breast?	Yes, No
	Is there a mass in the central region of the breast?	Yes, No
Mass Morphology	What is the shape of the mass in the left breast?	round, oval, lobular, irregular
Mass Morphology	What is the margin of the mass in the left breast?	circumscribed, irregular, spiculated

Appendix

Table 7: Lesion-level question templates used for MRI VQA generation.

Category	Question Template	Options
Presence & Surgical Procedure	Is there a pathological lesion identified in the left breast?	Yes, No
	Is there a pathological lesion identified in the right breast?	Yes, No
	What surgical procedure was performed on the left breast?	Lumpectomy, Mastectomy, Other
	What surgical procedure was performed on the right breast?	Lumpectomy, Mastectomy, Other
Tumor Characteristics	Is the maximum pathological tumor size larger than 2 cm?	Yes, No
Tumor Characteristics	What is the histological grade of the tumor?	1, 2, 3, Unknown

Appendix

Table 8: Pathology question templates used for pathology report VQA generation.

Question Template
Describe the overall fibroglandular tissue composition and background parenchymal enhancement of both breasts.
Provide a comprehensive description of the mass identified in the breast MRI.
Describe the imaging characteristics of the non-mass enhancement in the breast.
Describe the appearance of axillary and internal mammary lymph nodes.
Provide the final BI-RADS assessment and overall diagnostic impression.

Appendix

Table 9: Free-text diagnostic question templates used for holistic clinical reasoning.

Task	Fact.	Clin.	Overall	Pass (%)
Global background	2.94	2.92	2.93	99.2
Lesion localization	2.91	2.89	2.90	98.6
Morphology	2.86	2.83	2.84	97.5
Functional reasoning	2.79	2.75	2.77	96.2
Local invasion/nodal	2.88	2.86	2.87	97.9
Holistic diagnosis	2.82	2.79	2.80	96.8

Appendix

Table 10: Quality-control summary for generated QA pairs. Fact., Clin., and Overall denote factual consistency, clinical validity, and overall report support on a 0–3 scale. Pass rate denotes the percentage of expert-audited QA pairs that pass answer–evidence checking.

Task Name	Exact Keys	Prefixes
Global Background Assessment	breast_level.breast_symmetry	breast_level.FGT., breast_level.BPE., breast_level.post_surgical_changes.
Lesion Detection & Localization	mass, non_mass_enhancement, non_enhancing_lesion, mass.count, non_mass_enhancement.count, non_enhancing_lesion.count	mass.location., non_mass_enhancement.location., non_enhancing_lesion.location.
Morphological Characterization	non_mass_enhancement.distribution, non_enhancing_lesion.lesion_type_detail	mass.size_mm., mass.morphology., non_mass_enhancement.size_mm., non_enhancing_lesion.size_mm.
Multi-Modal Functional Reasoning	non_mass_enhancement. internal_enhancement_pattern	mass.signal_characteristics., non_mass_enhancement.signal_characteristics., non_enhancing_lesion.signal_characteristics., mass.internal_enhancement.
Local Invasion & Nodal Reasoning	-	breast_level.associated_features., breast_level.lymph_nodes.
Holistic Diagnosis	-	breast_level.BI-RADS., breast_level.management.

Appendix

Table 11: Key–task mapping rules used to organize generated questions into clinical-workflow based tasks.

Panel of Table 20 — List of algorithms 1 Greedy Prototype Selection (GPS)

Setting	Base VLM	GBA		LDL		Morph.		MMFR		LINR		HD		PP		Avg.
Setting	Base VLM	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.
SeVeR-3B	Qwen2.5-VL-3B	88.90	99.55	77.78	99.80	56.89	97.32	79.56	99.22	76.85	97.60	36.02	97.48	75.47	99.23	70.21	98.60
SeVeR-4B	Qwen3-VL-4B	89.82	99.68	78.48	99.87	57.45	97.46	80.11	99.43	77.42	97.77	36.53	97.59	74.18	99.10	70.57	98.70
SeVeR-8B	Qwen3-VL-8B	91.08	99.74	80.14	99.90	59.02	97.68	81.65	99.55	79.06	97.96	38.14	97.87	75.82	99.24	72.13	98.85

Appendix

Table 12: Full cross-model scaling on BreMRIs-VQA. Task abbreviations: GBA = Global Background Assessment, LDL = Lesion Detection and Localization, Morph. = Morphological Characterization, MMFR = Multi-Modal Functional Reasoning, LINR = Local Invasion and Nodal Reasoning, HD = Holistic Diagnosis, PP = Pathology Prediction. Avg. denotes the macro-average across the seven tasks.

Setting	GBA		LDL		Morph.		MMFR		LINR		HD		PP		Avg.
Setting	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.	Acc.	BERT.
Effect of $\tau_{v}$ (visual-affinity temperature).
$\tau_{v}$ =0.05	89.10	99.55	77.64	99.75	56.72	97.21	79.42	99.21	76.83	97.53	36.05	97.32	73.26	98.92	69.86	98.50
$\tau_{v}$ =0.10 †	89.82	99.68	78.48	99.87	57.45	97.46	80.11	99.43	77.42	97.77	36.53	97.59	74.18	99.10	70.57	98.70
$\tau_{v}$ =0.20	89.36	99.61	78.03	99.80	57.21	97.33	79.73	99.29	77.08	97.61	36.31	97.44	73.55	98.98	70.18	98.58
Effect of $k$ (prototype budget).

Appendix

Table 13: Hyperparameter sensitivity on BreMRIs-VQA. Each block varies one hyperparameter while keeping the others fixed at their default values, marked with

\dagger

. We report multiple-choice accuracy (Acc.) and free-text BERTScore (BERT). GBA = Global Background Assessment, LDL = Lesion Detection and Localization, Morph. = Morphological Characterization, MMFR = Multi-Modal Functional Reasoning, LINR = Local Invasion and Nodal Reasoning, HD = Holistic Diagnosis, PP = Pathology Prediction.

Type	Subtype	Multiple-choice						Free-text
		Merlin	M3D-L2	M3D-P3	CT-CHAT	RadFM	SeVeR	Merlin	M3D-L2	M3D-P3	CT-CHAT	RadFM	SeVeR
		7B	7B	4B	7B	13B	4B	7B	7B	4B	7B	13B	4B
Meas	Lesion Volume $\ast$	0.253	0.815	0.825	0.833	0.815	0.819	0.079	0.085	0.079	0.075	0.112	0.258
	Organ HU $\ast$	0.254	0.638	0.640	0.637	0.647	0.637	0.487	0.490	0.491	0.513	0.608	0.620
	Organ Volume $\ast$	0.262	0.747	0.754	0.750	0.755	0.748	0.526	0.535	0.528	0.549	0.583	0.685
	Average	0.256	0.733	0.740	0.740	0.739	0.735	0.364	0.370	0.366	0.379	0.434	0.521

Appendix

Table 14: Full subtype-level performance on DeepTumorVQA. Subtypes marked with

\ast

indicate free-text numerical answers evaluated using MRA (higher is better).

Metrics	E.D.	S.T.D.	L.T.D.	Medical Measurement			Image Observation			Anomaly Detection			Exposed tokens
Metrics	Acc.	Acc.	Acc.	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	Exposed tokens
Baseline Zs.	3.52	0	0.15	0.11	0.08	76.65	0.26	0.44	76.22	0.24	0.54	76.65	4608
SeVeR Phase1	22.94	10.54	21.12	10.47	20.78	91.23	4.74	17.94	86.5	5.36	17.28	86.24	512
Baseline Ft.	82.01	45.16	72.19	32.95	33.78	95.03	42.63	45.25	91.92	34.27	37.54	90.58	4608
w/o CaGA	75.73	41.70	66.66	30.43	31.19	87.75	39.36	41.78	84.88	31.64	34.66	83.64	512
$k$ =256	82.11	51.36	75.63	37.21	39.56	95.95	48.84	53.72	93.10	39.07	44.85	91.56	256

Appendix

Table 15: Full ablation study on 3D-RAD. We report task performance and the average number of exposed input visual tokens. E.D. = Existence Detection, S.T.D. = Static Temporal Diagnosis, L.T.D. = Longitudinal Temporal Diagnosis, Zs. = Zero-shot, Ft. = Fine-tune, Acc. = Accuracy, and BERT = BERTScore.

Panel of Figure 9 — Figure 9: Robustness to modality availability under different settings. SeVeR (solid) vs. w/o selective exposure (dashed) under six modality settings (S1–S6). (a) BERTScore, (b) BLEU, (c) ROUGE for free-text quality.

Task	Model	Acc.						BLEU
Task	Model	S1	S2	S3	S4	S5	S6	S1	S2	S3	S4	S5	S6
Global Background Assessment	w/o Proto.	89.82	89.82	83.10	87.78	79.97	89.43	95.66	95.90	72.76	95.43	96.49	96.63
	SeVeR	89.82	89.82	89.82	87.78	80.77	89.82	96.63	96.63	96.65	96.63	96.44	96.63
Lesion Detection & Localization	w/o Proto.	78.39	76.60	69.76	72.68	54.69	73.55	97.78	98.16	60.99	97.53	98.62	98.74
	SeVeR	78.39	76.60	80.21	72.67	54.80	78.48	98.77	98.77	98.62	98.76	98.56	98.61
Morphological Characterization	w/o Proto.	57.35	49.50	37.80	53.65	27.92	57.68	73.63	74.42	41.22	73.96	74.78	70.34

Appendix

Table 16: Full task performance under different modality settings (S1–S6). Top : MC accuracy (Acc.) and free-text BLEU. Bottom : free-text ROUGE and BERTScore.

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding
May 19, 2026Mohammad H. Abbasi, Favour Nerrise, Shaurnav Ghosh +12Healthcare 3D Medical Imaging
Stanford University · 1Stanford University, Stanford, CA, USA
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
Apr 17, 2026Lama Moukheiber, Caleb M. Yeung, Haotian Xue +33D Spatial Reasoning 3D Medical Imaging
Georgia Institute of Technology · Harvard University · Georgetown University

May 19, 2026J/K move · Enter open · S save

NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding

Mohammad H. Abbasi, Favour Nerrise, Shaurnav Ghosh, Ridvan Yesiloglu+11

Stanford University · 1Stanford University, Stanford, CA, USA

Healthcare 3D Medical Imaging Medical VQA Medical Image Grounding 3D VQA Medical Image Benchmarks+1

Open paper AskPDF arXiv

We present NeuroQA, a large-scale benchmark for visual question answering in 3D brain magnetic resonance imaging (MRI), with 56,953 QA pairs from 12,977 subjects across 12 datasets. It spans ages 5-104 and five clinical domains: Alzheimer's, Parkinson's, tumors, white matter disease, and neurodevelopment. Unlike prior medical Visual Question Answering (VQA) efforts that operate on 2D slices or rely on narrow diagnostic labels, NeuroQA pairs every item with a full 3D volume. It evaluates 11 clinically grounded reasoning skills across Yes/No, multiple-choice, and open-ended formats. Of the 203 templates, 131 are image-grounded (answerable from a 3-plane viewer) and 72 are image-informed (ground truth from quantitative volumetry or clinical instruments). To remove text-only shortcuts, we apply answer-distribution refinement, reducing closed-format text-only accuracy from

>

80% to 44.6%; image necessity is assessed separately through an image-grounding protocol released with the benchmark. A 38-rule deterministic pipeline and two rounds of expert review verify every QA pair against FreeSurfer measurements, metadata, or radiology report fields, with zero same-subject contradictions across templates. We conduct a clinician evaluation in which two clinicians independently assess 100 frozen test items on a three-plane viewer. On closed-format (Yes/No + multiple-choice) test-public items, the best zero-shot vision-language model and a supervised 3D CNN baseline reach 47.5% and 43.7% accuracy respectively, both below the 49.4% text-only majority-template floor. NeuroQA adopts a two-tier release with public QA pairs for open-access datasets and reproducible generation scripts for datasets restricted by data use agreements (DUAs), plus subject-level splits, a held-out private test set, and an online leaderboard.

Abstract

Medical VQA

Explore similar work

NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI