PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation
Authors: Jianyu Sun, Zhenxuan Zhang, Guang Yang, Peter J. Lally
Organizations: Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK
Radiology report generation has matured almost entirely on 2D chest radiographs, where the default route to better reports is a larger backbone or a pre-training one on medical data. We revisit that assumption on 3D multi-sequence brain MRI, a volumetric multi-disease regime, and find that the model is not the lever. Zero-shot medical and radiology vision-language models transfer poorly to brain MRI, with chest radiograph specialists failing most conspicuously, and five backbones fine-tuned identically across three model families and an order of magnitude in scale differ only marginally. What determines the quality of the report is the information injected into the prompt. We delegate perception to upstream 3D segmentation and classification, serialize their outputs into a structured fact sentence, and prompt a LoRA-adapted vision-language model with it; we call this PerFact. In a controlled study that fixes the backbone, data split, target reports, and adaptation while varying only the injected grounding, perception-derived facts outperform retrieved prior reports, retrieval becomes redundant once facts are present, and end-to-end predicted facts remain effective without any ground-truth annotation at inference. The residual gap between predicted and oracle facts is explained by the granularity of the facts rather than by the generator. Closed-ended visual question answering comes at no measurable cost to report quality, though the grounding source has little effect on it. On 3D brain MRI, grounding information, not model choice, is the dominant controllable factor in report quality.
Figures & tables
Figure 1: The information gap PerFact closes. This acute stroke study holds 496 axial slices and the prompt carries three of them, 0.6% of the study, with a fixed instruction. The fact sentence comes from an upstream model that reads all 496; the vision-language model still sees only the same three slices.
Figure 2: Injected facts correct a hallucinated location (case BraTS-GLI-00599). Left: four sequences of one slice, radiological orientation, reference segmentation outlined, with the fact given to PerFact. Right: verbatim output, … marking an elision, green agreeing with the reference and orange contradicting it. The lower block varies only the injected text. † A differential and a management plan rather than a report. ‡ A thoracic structure.
Figure 3: Overview of PerFact . Upstream 3D perception (top) turns a multi-contrast brain MRI study into a structured fact sentence (SF); the same sentence is the query for retrieval over training-split reports (CR) or a knowledge base (KR) (bottom). The shared encoder drawn here is the tumor multi-task network; stroke and white-matter hyperintensity each use a standalone binary segmenter instead. The fact sentence, generated report, and VQA pairs are verbatim system output for one glioma case.
Method
Injected grounding
CE metrics
NLG metrics
Closed-ended VQA
PREC
REC
F-1
BLEU-4
METEOR
ROUGE-L
Acc
F-1
AUROC
Zero-shot
none, no adaptation
0.541
0.480
0.509
2.68
20.32
15.52
0.567
0.352
0.530
SFT
none
0.671
0.619
0.644
25.83
43.71
49.85
0.650
0.567
0.697
+ KR
knowledge
0.682
0.601
0.639
25.98
43.96
51.31
0.644
0.583
0.701
+ CR
prior case reports
0.701
0.736
0.718
28.15
46.70
51.68
0.683
0.488
0.701
+ SF (PerFact)
predicted facts
0.771
0.779
0.775
29.52
47.30
53.55
0.693
0.535
0.720
Table 1: Grounding source is the lever (report n=130 , VQA n=1549 ). Expands the Qwen2.5-VL block of Table 3 across eight grounding conditions: + KR retrieved knowledge, + CR retrieved prior case reports, + SF predicted facts, + SF ∗ ground-truth facts; only the injected information differs. Every cell is a single run, read against the seed band of the Experimental Setup. Best and second best among deployable methods; + SF ∗ is an upper bound, not a competitor.
Method
LLM
Size
CE metrics
NLG metrics
Closed-ended VQA
PREC
REC
F-1
BLEU-4
METEOR
ROUGE-L
Acc
F-1
AUROC
Qwen2.5-VL
Qwen2.5
3B
0.762
0.782
0.772
29.67
47.46
53.86
0.618
0.570
0.683
Qwen2.5-VL
Qwen2.5
7B
0.759
0.793
0.775
30.42
48.63
54.14
0.653
0.564
0.702
Qwen2.5-VL
Qwen2.5
32B
0.766
0.782
0.774
29.90
47.75
53.60
0.640
0.582
0.715
LLaVA-1.5
Vicuna
7B
0.750
0.786
0.767
29.99
47.56
53.64
0.662
0.494
0.685
InternVL3
Qwen2.5
8B
0.754
0.811
0.782
29.80
47.90
53.44
0.675
0.603
0.714
Table 2: Fine-tuned backbones under identical PerFact grounding , differing only in the backbone; every column is a mean over the same three seeds. Best and second best in each column.
Method
LLM
Size
CE metrics
NLG metrics
Closed-ended VQA
PREC
REC
F-1
BLEU-4
METEOR
ROUGE-L
Acc
F-1
AUROC
HuatuoGPT-Vision
Qwen2
7B
0.544
0.346
0.423
2.87
17.46
14.43
0.550
0.532
0.631
RadFM
MedLLaMA
14B
0.486
0.122
0.195
2.66
11.05
15.88
0.566
0.106
0.438
LLaVA-Med-1.5
Mistral
7B
0.503
0.090
0.153
0.70
8.13
11.84
0.394
0.525
0.547
LLaVA-Rad
Vicuna
7B
0.500
0.108
0.178
1.15
8.33
11.69
0.467
0.482
0.564
MAIRA-2
Vicuna
6.9B
0.455
0.041
0.075
1.12
7.79
12.39
–
–
–
Table 3: Report generation and closed-ended VQA on brain MRI ( n=130 reports, n=1549 questions). Blocks: medical-pretrained VLMs, chest radiology report generators (both zero-shot), then ours with one un-adapted anchor per backbone; Table 1 walks the intermediate conditions. LLaVA-Rad’s report F-1 is inflated by laterality words coinciding with brain entities; its recall of 0.108 is the honest figure. Answering no throughout scores 0.655 on the VQA bank, above every zero-shot model here, so AUROC is the column to read. Dashes: MAIRA-2 exposes no question-answering interface. Best and second best among deployable rows; zero-shot and Qwen2.5-VL cells are single runs, the InternVL3 PerFact row a three-seed mean.
Figure 4: PerFact against released models on all nine metrics (zero-shot, brain MRI). The general medical model and the two chest radiology report generators all collapse, most sharply on the NLG axes, while PerFact traces a full profile.
Figure 5: t-SNE of clinical entity vectors against the reference reports (brain MRI, n=130 each). All 1,170 vectors share one embedding, so the reference distribution is identical in every panel. Shading outlines each cloud and titles give the median Jaccard distance dJ . All panels but the last are zero-shot, and only PerFact’s outline coincides with the reference’s. The cluster left of center is the all-empty vector: 86 of LLaVA-Med-1.5’s reports name no recognized brain finding, against 3 of PerFact’s.
Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed Brain3D, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, Brain3D is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility
Mariano Barone, Francesco Di Serio, Giuseppe Riccio +4
University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy
Acute ischemic stroke (AIS) requires time-critical decision-making, where inaccurate interpretation of neuroimaging findings can lead to irreversible disability. Diffusion-weighted imaging (DWI) and apparent diffusion coefficient (ADC) maps from magnetic resonance imaging (MRI) are central to detecting acute infarction, yet generating factually reliable radiology reports directly from 3D MRI remains challenging due to the difficulty of learning robust cross-modal alignments between volumetric images and clinical text. We propose paired image-domain retrieval and text-domain augmentation (PIRTA), a retrieval-augmented generation framework that improves report factuality by avoiding explicit image-text alignment. PIRTA retrieves clinically similar 3D DWI/ADC volumes using a pretrained 3D vision encoder and leverages their paired clinician-authored reports to ground large language model (LLM)-based report generation. Experiments on multi-institutional in-house data, a held-out external privacy-preserving cohort, and the public ISLES benchmark demonstrate that PIRTA achieves strong image-domain retrieval performance and consistently improves ischemic-territory accuracy, a clinically grounded surrogate for report factuality, compared to direct image-to-text baselines. These results indicate that retrieval-grounded generation provides a scalable and reliable paradigm for producing factually consistent radiology reports from complex 3D brain MRI. Source code is available at https://github.com/jhlee0619/PIRTA.
Junhyeok Lee, Yujin Oh, Dahyoun Lee +9
Cancer Biology, Seoul National University College of Medicine, Korea · Radiology, Massachusetts General Hospital and Harvard Medical School, USA · Biomedical Systems Informatics, Yonsei University College of Medicine, Korea +10
Neuroradiologists rarely read a brain MRI in isolation, yet automated brain-MRI report generation has been built almost entirely for single studies. Temporal analysis has been explored on chest radiography and chest CT, but to our knowledge, longitudinal reporting for brain MRI, where interval change is often subtle and spatially distributed, remains unaddressed. We present BrainDiff, the first longitudinal vision-language system for brain MRI. BrainDiff outperforms both frontier general-purpose and single-study neuroimaging models on the same patient pairs. Moreover, BrainDiff retains 91% of internal RadGraph-XL entity+relation F1 (rg_er) on an external, cross-hospital cohort. Beyond the system, we contribute three analyses. First, we identify two independent grounding levers: a counterfactual objective with prior-report dropout, which increases measured image reliance by ~47%, and a staged curriculum. Together, these interventions raise image reliance 2.5-fold from the baseline. Second, we provide a factorial over prior-report availability and image identity, isolating a visual contribution of +0.0387 rg_er, which grows when the prior report is withheld. Third, a cheap change-decodability test for candidate backbones shows that interval change is decodable far more weakly than single-study pathology (0.60 vs. 0.77 AUROC). Code is publicly available at https://github.com/jhuldr/BrainDiff.
Krish Patel, Peirong Liu
Department of Electrical and Computer Engineering, Data Science and AI Institute, Johns Hopkins University