Less Is More? Selective Visual Attention to High-Importance Regions for Multimodal Radiology Summarization

Mst. Fahmida Sultana Naznin; Adnan Ibney Faruq; Mushfiqur Rahman; Niloy Kumar Mondal; Md. Mehedi Hasan Shawon; Md Rakibul Hasan

cs.CVMar 31, 2026

Less Is More? Selective Visual Attention to High-Importance Regions for Multimodal Radiology Summarization

Authors: Mst. Fahmida Sultana Naznin, Adnan Ibney Faruq, Mushfiqur Rahman, Niloy Kumar Mondal, Md. Mehedi Hasan Shawon, Md Rakibul Hasan

Organizations: Bangladesh University of Engineering and Technology, Dhaka 1000, Bangladesh · BRAC University, Dhaka 1212, Bangladesh · Curtin University, Bentley, WA 6102, Australia

Abstract

Automated radiology report summarization aims to distill verbose findings into concise clinical impressions, but existing multimodal models often struggle with visual noise and fail to meaningfully improve over strong text-only baselines in the FINDINGS $\to$ IMPRESSION transformation. We challenge two prevailing assumptions: (1) that more visual input is always better, and (2) that multimodal models add limited value when findings already contain rich image-derived detail. Through controlled ablations on MIMIC-CXR benchmark, we show that selectively focusing on pathology-relevant visual patches rather than full images yields substantially better performance. We introduce ViTAS, Visual-Text Attention Summarizer, a multi-stage pipeline that combines ensemble-guided MedSAM2 lung segmentation, bidirectional cross-attention for multi-view fusion, Shapley-guided adaptive patch clustering, and hierarchical visual tokenization feeding a ViT. ViTAS achieves SOTA results with 29.25% BLEU-4 and 69.83% ROUGE-L, improved factual alignment in qualitative analysis, and the highest expert-rated human evaluation scores. Our findings demonstrate that less but more relevant visual input is not only sufficient but superior for multimodal radiology summarization.

Figures & tables

Figure 1: Overview of the ViTAS pipeline: Chest X-rays (frontal and lateral) are lung-segmented using MedSAM2, then fused via a dual SwinV2 with cross-attention. Interpretability modules identify important regions, which are selectively tokenized and combined with text embeddings in a T5 decoder to generate the final clinical impression.

Figure 2: Ensemble-guided MedSAM2 lung segmentation pipeline. A reference box generates five shifted bounding boxes. MedSAM2 produces binary masks which are unioned into the final lung ROI mask. The segmented image retains only pulmonary fields.

Figure 3: Dual Swin Transformer V2 with bidirectional mid-fusion cross-attention. Frontal and lateral views are processed by separate backbones, exchange information via cross-attention, and their global features are fused before classification.

Text-Only Baselines
Model	B-4	R-L	BScore	CBert	RadG
BioBART	19.71	43.68	90.66	58.84	36.68	45.33
T5	21.83	44.61	91.45	38.49	38.26	46.79
Full-Image Multimodal Models (Single vs Multi-View)
ViT + T5 (Frontal View)	17.71	52.30	92.05	59.90	44.70	53.45
ViT + T5 (Lateral View)	20.30	56.42	92.74	59.06	49.29	56.89

Table 1: Ablation study on MIMIC-CXR summarization evaluating (1) text-only models, (2) full-image multimodal inputs (single vs multi-view), (3) ROI-guided visual context, and selective patch-based visual evidence. Results show that selecting salient visual evidence is more effective than simply increasing the amount of visual input. The evaluation metrics include BLEU-4 (B-4), ROUGE-L (R-L), BERTScore (BScore), CheXbert score (CBert), and RadGraph score (RadG).

Text-Only Baselines
Model	B-4	R-L	BScore	CBert	RadG
PG Sim et al. (2025)	–	36.84	–	54.44	26.72
BART Sim et al. (2025)	–	47.00	–	65.39	40.90
GSum Sim et al. (2025)	–	42.98	–	59.60	35.18
WGSum Sim et al. (2025)	–	41.55	–	58.92	31.17
Llama-3.1-8B Zhao et al. (2025)	11.67	32.46	54.56	69.03	29.16

Table 2: Performance comparison of SOTA models on the MIMIC-CXR test set. Text-only results are shown at the top, followed by multimodal models.

Figure 4: Attention-driven patch selection pipeline (frontal top row, lateral bottom row): original X-ray, Swin V2 32 $\times$ 32 heatmap (downsampled from 64 $\times$ 64), overlay on MedSAM2-segmented lungs, DBSCAN clusters (15 frontal, 8 lateral), projected ViT 14 $\times$ 14 heatmap, and final selected clusters (8 frontal, 5 lateral). Only these pathology-dominant patches feed the multimodal T5 decoder.

Class	AP	PA	L	LL
Atelectasis	0.7191	0.7227	0.8340	0.8128
Cardiomegaly	0.8252	0.8162	0.9035	0.8507
Consolidation	0.7509	0.7452	0.8485	0.8506
Edema	0.8689	0.8713	0.9348	0.9128
No Finding	0.7854	0.7857	0.8308	0.8241
Pleural Effusion	0.8666	0.8650	0.9315	0.9568

Table 3: Per-class AUROC across different chest X-ray projection views. AP, PA, L, and LL indicate Anteroposterior, Posteroanterior, Lateral, and Left Lateral views.

Figure 5: Qualitative analysis of left lung opacification. Ground truth shows increasing opacification. Full-image and ROI models detect pneumonia with suggested follow-up, while the ViTAS ROI-patch model closely matches ground truth, capturing pneumonia and possible lymphangitic spread with suggestions.

Model	Readability	Factual Correctness	Informativeness	Redundancy	Completeness
Full-Image Multimodal Model	4.11	4.22	3.78	4.00	3.67
ROI (Segmented Regions)	4.15	4.18	3.85	4.05	3.72
ViTAS (ROI Patches)	3.92	5.00	4.95	3.88	5.00

Table 4: Human evaluation results based on summarization quality. Higher scores indicate better performance. Scores are reported as floats with two decimal points.

Radiology VLMs

Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.

See all 109 papers

Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context
Jul 7, 2026Suneeta Mall, Vladimir Nekrasov, Ashnil Kumar +5Radiology Report Generation Radiology VLMs
Vision-language models for chest radiography do not always need the image
Jun 16, 2026Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams +4VLM Evaluation Medical VQA
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +1
RadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings
Mar 24, 2026Sitong Zhou, Meliha Yetisgen, Mari OstendorfText Summarization Clinical Information Extraction
University of Washington

Jul 7, 2026J/K move · Enter open · S save

Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context

Suneeta Mall, Vladimir Nekrasov, Ashnil Kumar, Sajith Karunasena+4

Radiology Report Generation Radiology VLMs Multimodal Large Language Models Radiology Report Evaluation

Open paper AskPDF arXiv

Imaging demand is growing faster than the radiology workforce can expand, and reporting backlogs cannot be resolved through training and recruitment alone. The most direct opportunity is reducing the time and effort radiologists spend producing reports, a task that requires interpreting images, integrating clinical history and prior studies, and drafting structured findings. We present Harrison.Rad 1.5 (HR1.5), a radiology-specific multimodal large language model that accepts interleaved text and visual inputs and generates structured and unstructured text across plain-film radiology, spanning computed radiography, chest, musculoskeletal, abdominal, spine, and pelvic x-rays, and mammography. HR1.5 is trained through a three-stage pipeline: domain adaptation of a base language model on radiology reports, contrastive vision-encoder training with curriculum-based hard negatives on ~6 million image-report instances, and visual-question-answering fine-tuning on multi-turn conversations. We evaluate it with a Findings-Diagnosis scoring framework that extends RadGraph-XL entity extraction with ontology-based synonym matching and polarity-contradiction detection, benchmarked on RadBench, a simulated FRCR 2B Short Case examination scored against Angoff-method thresholds, ReXGradient, and internal multi-modality datasets. HR1.5 is the only system evaluated to meet the simulated FRCR passing standard and achieves the highest accuracy on closed-format clinical questions, across anatomical regions, on internal multi-body-part and mammography reporting, and on the primary clinically-aligned score for public chest reporting. We further examine explainability and model behaviour, including question-sensitive Grad-CAM heatmaps, attention analysis, and confidence estimation, to support responsible future evaluation toward clinical use, and a framework for clinically grounded assessment of report quality.

Abstract

Radiology VLMs

Explore similar work

Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context

Vision-language models for chest radiography do not always need the image

RadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings