PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation

Jianyu Sun; Zhenxuan Zhang; Guang Yang; Peter J. Lally

cs.CVAug 18, 2026

PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation

Authors: Jianyu Sun, Zhenxuan Zhang, Guang Yang, Peter J. Lally

Organizations: Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK

Abstract

Radiology report generation has matured almost entirely on 2D chest radiographs, where the default route to better reports is a larger backbone or a pre-training one on medical data. We revisit that assumption on 3D multi-sequence brain MRI, a volumetric multi-disease regime, and find that the model is not the lever. Zero-shot medical and radiology vision-language models transfer poorly to brain MRI, with chest radiograph specialists failing most conspicuously, and five backbones fine-tuned identically across three model families and an order of magnitude in scale differ only marginally. What determines the quality of the report is the information injected into the prompt. We delegate perception to upstream 3D segmentation and classification, serialize their outputs into a structured fact sentence, and prompt a LoRA-adapted vision-language model with it; we call this PerFact. In a controlled study that fixes the backbone, data split, target reports, and adaptation while varying only the injected grounding, perception-derived facts outperform retrieved prior reports, retrieval becomes redundant once facts are present, and end-to-end predicted facts remain effective without any ground-truth annotation at inference. The residual gap between predicted and oracle facts is explained by the granularity of the facts rather than by the generator. Closed-ended visual question answering comes at no measurable cost to report quality, though the grounding source has little effect on it. On 3D brain MRI, grounding information, not model choice, is the dominant controllable factor in report quality.

Figures & tables

Panel of Figure 1 — Figure 1: The information gap PerFact closes. This acute stroke study holds 496 axial slices and the prompt carries three of them, 0.6% of the study, with a fixed instruction. The fact sentence comes from an upstream model that reads all 496; the vision-language model still sees only the same three slices.

Figure 2: Injected facts correct a hallucinated location (case BraTS-GLI-00599). Left: four sequences of one slice, radiological orientation, reference segmentation outlined, with the fact given to PerFact. Right: verbatim output, … marking an elision, green agreeing with the reference and orange contradicting it. The lower block varies only the injected text. $\dagger$ A differential and a management plan rather than a report. $\ddagger$ A thoracic structure.

Figure 3: Overview of PerFact . Upstream 3D perception (top) turns a multi-contrast brain MRI study into a structured fact sentence (SF); the same sentence is the query for retrieval over training-split reports (CR) or a knowledge base (KR) (bottom). The shared encoder drawn here is the tumor multi-task network; stroke and white-matter hyperintensity each use a standalone binary segmenter instead. The fact sentence, generated report, and VQA pairs are verbatim system output for one glioma case.

Method	Injected grounding	CE metrics			NLG metrics			Closed-ended VQA
Method	Injected grounding	PREC	REC	F-1	BLEU-4	METEOR	ROUGE-L	Acc	F-1	AUROC
Zero-shot	none, no adaptation	0.541	0.480	0.509	2.68	20.32	15.52	0.567	0.352	0.530
SFT	none	0.671	0.619	0.644	25.83	43.71	49.85	0.650	0.567	0.697
$+$ KR	knowledge	0.682	0.601	0.639	25.98	43.96	51.31	0.644	0.583	0.701
$+$ CR	prior case reports	0.701	0.736	0.718	28.15	46.70	51.68	0.683	0.488	0.701
$+$ SF (PerFact)	predicted facts	0.771	0.779	0.775	29.52	47.30	53.55	0.693	0.535	0.720

Table 1: Grounding source is the lever (report

n{=}130

, VQA

n{=}1549

). Expands the Qwen2.5-VL block of Table 3 across eight grounding conditions:

+

KR retrieved knowledge,

+

CR retrieved prior case reports,

+

SF predicted facts,

+

SF ∗ ground-truth facts; only the injected information differs. Every cell is a single run, read against the seed band of the Experimental Setup. Best and second best among deployable methods;

+

SF ∗ is an upper bound, not a competitor.

Method	LLM	Size	CE metrics			NLG metrics			Closed-ended VQA
Method	LLM	Size	PREC	REC	F-1	BLEU-4	METEOR	ROUGE-L	Acc	F-1	AUROC
Qwen2.5-VL	Qwen2.5	3B	0.762	0.782	0.772	29.67	47.46	53.86	0.618	0.570	0.683
Qwen2.5-VL	Qwen2.5	7B	0.759	0.793	0.775	30.42	48.63	54.14	0.653	0.564	0.702
Qwen2.5-VL	Qwen2.5	32B	0.766	0.782	0.774	29.90	47.75	53.60	0.640	0.582	0.715
LLaVA-1.5	Vicuna	7B	0.750	0.786	0.767	29.99	47.56	53.64	0.662	0.494	0.685
InternVL3	Qwen2.5	8B	0.754	0.811	0.782	29.80	47.90	53.44	0.675	0.603	0.714

Table 2: Fine-tuned backbones under identical PerFact grounding , differing only in the backbone; every column is a mean over the same three seeds. Best and second best in each column.

Method	LLM	Size	CE metrics			NLG metrics			Closed-ended VQA
Method	LLM	Size	PREC	REC	F-1	BLEU-4	METEOR	ROUGE-L	Acc	F-1	AUROC
HuatuoGPT-Vision	Qwen2	7B	0.544	0.346	0.423	2.87	17.46	14.43	0.550	0.532	0.631
RadFM	MedLLaMA	14B	0.486	0.122	0.195	2.66	11.05	15.88	0.566	0.106	0.438
LLaVA-Med-1.5	Mistral	7B	0.503	0.090	0.153	0.70	8.13	11.84	0.394	0.525	0.547
LLaVA-Rad	Vicuna	7B	0.500	0.108	0.178	1.15	8.33	11.69	0.467	0.482	0.564
MAIRA-2	Vicuna	6.9B	0.455	0.041	0.075	1.12	7.79	12.39	–	–	–

Table 3: Report generation and closed-ended VQA on brain MRI (

n{=}130

reports,

n{=}1549

questions). Blocks: medical-pretrained VLMs, chest radiology report generators (both zero-shot), then ours with one un-adapted anchor per backbone; Table 1 walks the intermediate conditions. LLaVA-Rad’s report F-1 is inflated by laterality words coinciding with brain entities; its recall of 0.108 is the honest figure. Answering no throughout scores 0.655 on the VQA bank, above every zero-shot model here, so AUROC is the column to read. Dashes: MAIRA-2 exposes no question-answering interface. Best and second best among deployable rows; zero-shot and Qwen2.5-VL cells are single runs, the InternVL3 PerFact row a three-seed mean.

Figure 4: PerFact against released models on all nine metrics (zero-shot, brain MRI). The general medical model and the two chest radiology report generators all collapse, most sharply on the NLG axes, while PerFact traces a full profile.

Figure 5: t-SNE of clinical entity vectors against the reference reports (brain MRI,

n{=}130

each). All 1,170 vectors share one embedding, so the reference distribution is identical in every panel. Shading outlines each cloud and titles give the median Jaccard distance

d_{J}

. All panels but the last are zero-shot, and only PerFact’s outline coincides with the reference’s. The cluster left of center is the all-empty vector: 86 of LLaVA-Med-1.5’s reports name no recognized brain finding, against 3 of PerFact’s.

Explore similar work

Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
Feb 25, 2026Mariano Barone, Francesco Di Serio, Giuseppe Riccio +4Radiology Report Generation Medical VLMs
University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy
Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation
Nov 23, 2024Junhyeok Lee, Yujin Oh, Dahyoun Lee +9Radiology Report Generation Retrieval-Augmented Generation
Cancer Biology, Seoul National University College of Medicine, Korea · Radiology, Massachusetts General Hospital and Harvard Medical School, USA · Biomedical Systems Informatics, Yonsei University College of Medicine, Korea +10
BrainDiff: Longitudinal Report Generation for Multimodal Brain MRI
Sep 1, 2026Krish Patel, Peirong LiuRadiology Report Generation Longitudinal Medical Image Analysis
Department of Electrical and Computer Engineering, Data Science and AI Institute, Johns Hopkins University

Feb 25, 2026J/K move · Enter open · S save

Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

Mariano Barone, Francesco Di Serio, Giuseppe Riccio, Antonio Romano+3

University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy

Radiology Report Generation Medical VLMs 3D VLMs 3D ViTs

Open paper AskPDF arXiv

Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed Brain3D, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, Brain3D is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility

Abstract

Explore similar work

Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation

BrainDiff: Longitudinal Report Generation for Multimodal Brain MRI