Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks

Jonghun Kim; Sinyoung Ra; Hyunjin Park

cs.CVApr 3, 2026

Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks

Authors: Jonghun Kim, Sinyoung Ra, Hyunjin Park

Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea · Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Korea

Abstract

LLMs have demonstrated remarkable capabilities in linguistic reasoning and are increasingly adept at vision-language tasks. The integration of image tokens into transformers has enabled direct visual input and output, advancing research from image-to-text descriptions to text-to-image generation. However, simple text-to-image generation holds limited clinical utility. In medical imaging, tasks such as image segmentation for localizing pathologies or image translation for reconstructing missing sequences have much greater clinical importance. Despite this, integrating these diverse, clinically relevant tasks within a single, versatile language model remains unexplored. Our method, LLaBIT (Large Language Model for Brain Image Translation), extends the visual reasoning of LLMs to these clinically meaningful tasks in the brain MRI domain. To mitigate the spatial information loss inherent in image tokenization, we incorporate a mechanism to reuse feature maps from the image encoder, minimizing data degradation. We also generate text data using LLMs with strict predefined instructions to augment limited image-text paired data in brain MRI. We comprehensively evaluated our method on five brain MRI datasets across four distinct tasks: report generation, visual question answering, image segmentation, and image translation. Our model not only demonstrated superior performance across all tasks but also outperformed specialized, task-specific models in direct comparisons, highlighting its efficacy and versatility

Figures & tables

Figure 1: Example of LLaBIT performing versatile tasks on brain MR images . LLaBIT supports report generation and image-to-image tasks.

Figure 2: Text data generation with LLMs on a dataset with only images. Images and captions are processed by LLMs with strict predefined instructions and few-shot samples selected by clinicians to generate reports and VQA results. The output of each model is accepted or rejected using GPT 4o and the final report and VQA are regenerated based on this feedback.

Figure 3: Instruction tuning pipeline. Both text and images are tokenized and fed into the LLM, which can generate either text or image tokens as output. The LLM’s vocabulary is extended to include image tokens in addition to text tokens. The instruction is provided to the LLM as text tokens. The image is converted into quantized tokens using a VQ encoder and fed into the LLM along with an <input> token. These image tokens are added to the LLM’s vocabulary as discrete values, similar to text tokens. For image translation, the output is generated with an <output> token, while for segmentation, the output is generated with a <seg> token.

Figure 4: Fine-tuning of VQ-GAN with zero skip connection . The skip connection is fine-tuned, while freezing the image encoder and decoder. A zero convolution block is adopted using a BiomedCLIP text encoder and prompt tuning to flexibly adapt to the target.

Figure 5: Loss functions for image-to-image tasks. (a) The translation task is trained using reconstruction loss. (b) The segmentation task is trained using Dice loss with an additional layer.

Figure 6: Examples of report generation and VQA from various MLLMs . The highlighted text reflects the unique features of each image. For comparison, each image displays its modality, plane, and abnormality information at the top. Top: an example of report generation; Bottom: an example of VQA.

Report Generation	BraTS2021			BraTS2023-MEN			ATLAS 2.0			IXI
Report Generation	R.	M.	B.	R.	M.	B.	R.	M.	B.	R.	M.	B.
LLaVA-Med	7.19	12.37	85.52	6.26	11.38	85.43	7.15	14.37	86.74	9.25	19.70	87.69
LLaVA 1.6	20.66	19.98	84.24	21.48	21.10	84.34	25.05	26.00	86.40	28.56	30.59	87.80
LLama 3.2 11B	15.78	15.32	83.10	16.43	16.52	83.15	16.62	15.20	83.07	17.12	16.04	83.32
Gemini 1.5 8B	22.04	17.69	81.94	22.01	17.28	81.67	21.63	17.66	81.90	22.65	18.30	82.06
GPT 4o-mini	17.42	15.86	81.55	16.13	14.99	81.39	18.97	18.23	82.02	22.28	20.27	82.73

Table 1: Performance on image-to-text tasks . Top: Report generation performance evaluated across four datasets. Higher scores on the ROUGE, METEOR, and BERT-F1 metrics indicate better performance. Bottom: VQA performance on four datasets. "Mod." represents modality and "Abno." represents abnormality, with each value representing accuracy. Bold denotes the best performance, and * indicates a significant difference between ours and baselines (

p

-value

\ <0.05

Image Segmentation	BraTS2021				BraTS2023-MEN				UPENN-GBM				ATLAS 2.0
Image Segmentation	WT.	ET.	TC.	Avg.	WT.	ET.	TC.	Avg.	WT.	ET.	TC.	Avg.	Stroke
M3D (Pretrained)	0.025	0.024	0.010	0.019	0.013	0.008	0.008	0.010	0.041	0.035	0.016	0.030	0.010
M3D (Fine-tuned)	0.608	0.450	0.342	0.466	0.275	0.231	0.101	0.202	0.619	0.411	0.322	0.450	0.057
LLaBIT (Stage I)	0.653	0.445	0.389	0.495	0.492	0.373	0.235	0.366	0.645	0.439	0.318	0.467	0.269
LLaBIT (Stage II)	0.770 ∗	0.747 ∗	0.603 ∗	0.706 ∗	0.651 ∗	0.587 ∗	0.402 ∗	0.546 ∗	0.792 ∗	0.778 ∗	0.613 ∗	0.727 ∗	0.438 ∗

Table 2: Performance on image-to-image tasks . "WT." denotes whole tumor, "ET." denotes enhancing tumor, "TC." denotes tumor core, and "Avg." represents the average value. Each value is expressed as a dice coefficient. Bold indicates the best performance, underlined denotes the second best performance, and * significant difference between ours and baselines,

p

-value

\ <0.05

	Segmentation				T1 $\rightarrow$ T2
	WT.	ET.	TC.	Avg.	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
Stage I only	0.645	0.439	0.318	0.467	21.023	0.707	45.382	0.146
+ Skip Conv block	0.705	0.617	0.515	0.612	21.811	0.757	48.018	0.138
+ Zero init	0.725	0.653	0.534	0.637	22.312	0.778	45.723	0.124
+ Prompt tuning	0.773	0.762	0.608	0.714	22.617	0.801	45.047	0.117
+ BioMedCLIP	0.792	0.778	0.613	0.727	22.943	0.817	43.205	0.111

Table 4: Ablation study results on the UPENN-GBM .

Explore similar work

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology
Jul 16, 2026Sinyoung Ra, Jonghun Kim, Hyunjin ParkRadiology Report Generation 3D Medical Imaging
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea
Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
Feb 25, 2026Mariano Barone, Francesco Di Serio, Giuseppe Riccio +4Radiology Report Generation Medical VLMs
University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy
PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation
Aug 18, 2026Jianyu Sun, Zhenxuan Zhang, Guang Yang +1Radiology Report Generation Medical VLMs
Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK

Jul 16, 2026J/K move · Enter open · S save

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology

Sinyoung Ra, Jonghun Kim, Hyunjin Park

Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea

Radiology Report Generation 3D Medical Imaging Multimodal Instruction Tuning 3D VLMs

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) and their extension to vision-language models (VLMs) have made it easier to combine text and images for tasks such as report generation. Existing VLMs in medicine typically focus on 2D images (chest X-rays), and their extension to 3D imaging has been difficult because of the lack of paired 3D imaging-text data. Thus, we introduce a new method for creating a 3D image-text dataset for brain oncology using 3D MRI scans of glioma and meningioma cases. We use a cooperative system in which several LLMs work together to generate and check reports, ensuring that they are accurate and clear. By leveraging the new 3D MRI-text dataset, we further build a VLM that converts MRI scans into tokens and aligns them with text instructions. Our VLM performed better in report generation and visual question answering tasks than other 2D and 3D methods. Our method not only improves the quality of reports but also helps with better diagnosis and treatment in brain oncology.

Abstract

Explore similar work

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology

Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation