Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

Mariano Barone; Francesco Di Serio; Giuseppe Riccio; Antonio Romano; Marco Postiglione; Antonino Ferraro; Vincenzo Moscato

cs.CVFeb 25, 2026

Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

Authors: Mariano Barone, Francesco Di Serio, Giuseppe Riccio, Antonio Romano, Marco Postiglione, Antonino Ferraro, Vincenzo Moscato

Organizations: University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy

Abstract

Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed Brain3D, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, Brain3D is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility

Figures & tables

Fig. 1: Brain3D Architecture. A standardized MRI volume $X$ is processed by an inflated 3D Transformer encoder, producing $N$ volumetric patch tokens $Z_{\text{enc}}$ . These tokens are compressed via Vision Token Compression into a fixed set of $K=32$ tokens $Z_{\text{cmp}}$ . The compressed visual tokens are projected into the language embedding space ( $Z_{\text{proj}}$ ) and scaled to obtain conditioning tokens $Z_{\text{cond}}$ , which are prepended to the textual embeddings and used to guide autoregressive report generation by the causal LLM.

Fig. 2: Staged Training Strategy The framework employs a progressive three-phase alignment pipeline. Phase 1: Contrastive Image-Text Grounding aligns the 3D representations ( $Z_{vis}$ ) with report semantics ( $Z_{t}$ ) using a symmetric InfoNCE loss. Phase 2A: Projector Warmup performs supervised generation with a frozen LLM to stabilize the visual-language mapping. Phase 2B: Linguistic Adaptation fine-tunes the projector and LoRA adapters jointly to capture neuroradiology syntax. Legend: Modules marked with Ice ( ) are frozen; modules marked with Fire ( ) are trainable.

State-of-the-Art Baselines
Model	Natural Language Generation (NLG)						Semantics		Clinical Efficacy (F1)
Model	B-1	B-4	R-1	R-2	R-L	MTR	BERT	CIDEr	Lat	Anat	Path
Med3DVLM [ 6 ]	0.051	0.005	0.108	0.018	0.083	0.055	0.836	0.007	0.300	0.225	0.119
(3D Generalist)	[0.04, 0.06]	[0.00, 0.01]	[0.09, 0.13]	[0.01, 0.02]	[0.07, 0.10]	[0.05, 0.06]	[0.83, 0.84]	[0.00, 0.00]	[0.22, 0.40]	[0.15, 0.30]	[0.08, 0.18]
MedGemma 1.5 [ 3 ]	0.245	0.024	0.326	0.077	0.189	0.190	0.859	0.029	0.526	0.461	0.413
(2D Slice-based)	[0.22, 0.27]	[0.02, 0.03]	[0.30, 0.35]	[0.07, 0.09]	[0.17, 0.20]	[0.18, 0.20]	[0.86, 0.86]	[0.02, 0.03]	[0.45, 0.60]	[0.39, 0.53]	[0.36, 0.46]

TABLE I: Main Results & Ablation. Top: Comparison against Med3DVLM and MedGemma 1.5 4B baselines. Bottom: Evolution of our framework through training phases. Note the massive gain in Clinical Efficacy (F1) of our final model (Phase 2b) compared to the strong 2D baseline (MedGemma 1.5). Best results per column are in bold . Values: Mean (95% CI).

Qualitative Comparison of Generated Reports
			$<$ Ground Truth $>$ : The lesion area is in the frontal and parietal regions of the left hemisphere with mixed signal intensities of varying levels and speckled high-intensity signal areas. Edema is primarily concentrated in the peripheral regions of the left frontal and parietal lobes, indicating moderate to severe swelling of the tissues surrounding the lesions. Necrosis is observed in some lesion areas displaying low signal intensity interspersed within regions of high and isointense signals, showing a certain mottled appearance. Ventricular compression is mild deformation of the ventricles, suggesting pressure effects of the lesions on these regions.
			$<$ Brain3D (Ours) $>$ : The lesion area is in the left parietal and frontal lobes with mixed high-signal areas. Edema signals are mainly observed around these lesions, indicating significant edema presence affecting parts of both frontal and temporal regions as well as some portions within the parietal lobe. Necrosis may be present at low signal intensity or scattered throughout certain sections of the brain tissue affected by edema but shows no clear concentration. Ventricular compression effects on adjacent ventricles can occur due to pressure from surrounding tissues near the ventricular system.
			$<$ MedGemma Baseline $>$ : There is a large, irregular, mixed signal intensity mass centered in the right frontal lobe , extending into the right temporal lobe and involving the right lateral ventricle . There is significant mass effect on the right lateral ventricle , causing mild to moderate effacement of the adjacent sulci.
			$<$ Med3D-VLM $>$ : No intracranial hemorrhage. No evidence of acute infarction. No mass effect or midline shift. Normal grey-white matter differentiation .

TABLE II: Qualitative Comparison. Visual comparison of generated reports for a representative test sample. Our model identifies the lesion location and pathologies, whereas baselines hallucinate or fail.

Fig. 3: 3D LIME Attribution Maps. Volumetric grounding visualized via 3D LIME over SLIC supervoxels for a representative test case. Red regions indicates positive attribution (supporting the report), blue regions negative attribution. The tumor-bearing hemisphere is correctly highlighted; however, diffuse and partially contralateral supervoxels are also activated, suggesting reliance on both lesion-centered and global contextual patterns, potentially contributing to lateralization errors.

Explore similar work

PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation
Aug 18, 2026Jianyu Sun, Zhenxuan Zhang, Guang Yang +1Radiology Report Generation Medical VLMs
Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK
BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features
Feb 17, 2026Juampablo E. Heras Rivera, Dickson T. Chen, Tianyi Ren +4Radiology Report Generation Text Generation
University of Washington · University of Washington School of Medicine · Microsoft Health AI +1
Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology
Jul 16, 2026Sinyoung Ra, Jonghun Kim, Hyunjin ParkRadiology Report Generation 3D Medical Imaging
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea

Aug 18, 2026J/K move · Enter open · S save

PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation

Jianyu Sun, Zhenxuan Zhang, Guang Yang, Peter J. Lally

Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK

Radiology Report Generation Medical VLMs 3D Medical Imaging Medical Image Grounding

Open paper AskPDF arXiv

Radiology report generation has matured almost entirely on 2D chest radiographs, where the default route to better reports is a larger backbone or a pre-training one on medical data. We revisit that assumption on 3D multi-sequence brain MRI, a volumetric multi-disease regime, and find that the model is not the lever. Zero-shot medical and radiology vision-language models transfer poorly to brain MRI, with chest radiograph specialists failing most conspicuously, and five backbones fine-tuned identically across three model families and an order of magnitude in scale differ only marginally. What determines the quality of the report is the information injected into the prompt. We delegate perception to upstream 3D segmentation and classification, serialize their outputs into a structured fact sentence, and prompt a LoRA-adapted vision-language model with it; we call this PerFact. In a controlled study that fixes the backbone, data split, target reports, and adaptation while varying only the injected grounding, perception-derived facts outperform retrieved prior reports, retrieval becomes redundant once facts are present, and end-to-end predicted facts remain effective without any ground-truth annotation at inference. The residual gap between predicted and oracle facts is explained by the granularity of the facts rather than by the generator. Closed-ended visual question answering comes at no measurable cost to report quality, though the grounding source has little effect on it. On 3D brain MRI, grounding information, not model choice, is the dominant controllable factor in report quality.

Abstract

Explore similar work

PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation

BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology