Location-Aware Pretraining for Medical Difference Visual Question Answering

Denis Musinguzi; Caren Han; Prasenjit Mitra

cs.CVMar 5, 2026

Location-Aware Pretraining for Medical Difference Visual Question Answering

Authors: Denis Musinguzi, Caren Han, Prasenjit Mitra

Organizations: Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda · University of Melbourne, Melbourne, Australia.

Abstract

Differential medical VQA models compare multiple images to identify clinically meaningful changes and rely on vision encoders to capture fine-grained visual differences that reflect radiologists' comparative diagnostic workflows. However, vision encoders trained using standard contrastive or classification objectives often fail to capture the subtle variations needed to distinguish true disease progression from acquisition-related variability. To address this limitation, we introduce a location-aware pretraining framework that incorporates automatic referring expressions (AREF), grounded captioning (GCAP), and conditional automatic referring expressions (CAREF). These tasks promote the learning of fine-grained, spatially grounded visual representations. When integrated with a language model, our approach achieves state-of-the-art performance on medical difference VQA by accurately identifying and reasoning about clinically relevant changes in chest X-ray images.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of medical difference model architecture . The model consists of a frozen pretrained vision encoder, a vision adapter, and a transformer decoder.

Figure 2: Overview of pretraining model architecture . The model consists of a Siglip vision encoder and a transformer decoder. The vision encoder takes a chest X-ray image as input, produces visual tokens as cross-attention input to the transformer decoder. We adopt the following four tasks for pretraining: Cap, AREF, GCAP, and CAREF.

BLEU-4	METEOR	ROUGE-L	CIDEr
0.476	0.501	0.593	1.201

Table 1: Results showing the performance of the pre-trained model on the grounded captioning task.

Method	BLEU-4	METEOR	ROUGE-L	CIDEr	BertScore
Location Aware Pretraining (Ours)	0.594	0.425	0.747	2.997	0.972
Global Contrastive Pretraining	0.291	0.300	0.407	0.550	0.850
Regional Contrastive Pretraining	0.360	0.338	0.465	0.635	0.956
BLIP-2 ( Li et al., 2023b )	0.375	0.350	0.545	0.801	0.960
CapPa ( Tschannen et al., 2023 )	0.350	0.327	0.529	0.675	0.948
RG-AG ( Serra et al., 2025 )	0.551	0.384	0.668	2.198	0.965

Table 2: Results showing the comparative performance of various vision encoder pretraining methods, each combined with a GPT-2 decoder, on the Medical Difference VQA task. The best results are in in bold . Under the same training settings, our method outperforms the baselines and the state of the art approaches.

Case I	Case II	Case III
Reference Main	Reference Main	Reference Main

Q : What has changed compared to the reference image?	Q : What has changed compared to the reference image?	Q : What has changed compared to the reference image?
GT : the main image has additional findings of lung opacity , and atelectasis than the reference image.	GT : the main image has an additional finding of atelectasis than the reference image.	GT : the main image has additional findings of cardiomegaly, edema , atelectasis , and pleural effusion than the reference image. the main image is missing the findings of lung opacity , and pneumonia than the reference image.
Ours : the main image has additional findings of lung opacity , and atelectasis than the reference image.	Ours : the main image has an additional finding of atelectasis than the reference image.	Ours : the main image has additional findings of atelectasis , and pleural effusion than the reference image. the main image is missing the findings of lung opacity , and pneumonia than the reference image.
ReAL : the main image has additional findings of lung opacity , and atelectasis than the reference image.	ReAL : the main image has additional findings of pleural effusion , and atelectasis than the reference image.	ReAL : the main image has additional findings of atelectasis , and lung opacity than the reference image.

Table 3: Selected test cases to qualitatively compare the outputs of our approach with those of the baselines in Medical Diff VQA. Red indicates incorrect or missing predictions and green indicates correct predictions. The examples illustrate our model’s ability to identify differences between the two images and correctly attribute them to the appropriate image. However, there are instances where the model omits certain relevant features.

AREF	GCAP	CAREF	CAP	BLEU-4	METEOR	ROUGE-L	CIDEr
✓	✓	✓	✓	0.594	0.425	0.747	2.997
✗	✓	✓	✓	0.283	0.244	0.379	0.850
✓	✗	✓	✓	0.347	0.318	0.527	0.945
✓	✓	✗	✓	0.347	0.316	0.533	0.946
✗	✗	✗	✓	0.350	0.317	0.529	0.950

Table 4: Results of the ablation study when we removed some of the location aware pretraining tasks. The best results are in bold .

Image resolution	BLEU-4	METEOR	ROUGE_L
$224\times 224$	0.384	0.356	0.504
$448\times 448$	0.594	0.425	0.747

Table 5: Ablation study of the impact of different image resolutions on the medical difference VQA. The best results are in bold .

Parallel (%)	BLEU-4	METEOR	ROUGE-L
0	0.301	0.304	0.448
25	0.594	0.425	0.747
50	0.358	0.339	0.484

Table 6: Results showing the impact of the percentage of training samples that are masked during the vision encoder pretraining on downstream performance of the vision encoder. The best results are in bold .

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models
Jun 3, 2026Jialin Wu, Qianru Zhang, Georges El Fakhri +1Vision Foundation Model Adaptation Medical VQA
Yale Biomedical Imaging Institute New Haven, CT 06510
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
InViC: Intent-aware Visual Cues for Medical Visual Question Answering
Mar 17, 2026Zhisong Wang, Ziyang Chen, Zanting Ye +3Medical VQA Medical VLMs
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University

Jun 3, 2026J/K move · Enter open · S save

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

Jialin Wu, Qianru Zhang, Georges El Fakhri, Xiaofeng Liu

Yale Biomedical Imaging Institute New Haven, CT 06510

Vision Foundation Model Adaptation Medical VQA Explainable Medical Image Analysis

Open paper AskPDF arXiv

Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point. We propose an attention-guided encoder-decoder for this task with chest X-rays. Instead of conventional direct contrast, we propose to include a lightweight affine registration module to reduce nuisance motion by co-registering the current image to the reference image with a small registration regularizer. The registered image pair is fed into the image encoder, followed by a frozen DINO-based mask generator and a trainable adaptive mask generator to produce masks applied to the original image pairs. The masked image pairs are again fed into the image encoder and concatenated with text features as the input to a multimodal transformer-based decoder to generate final answers. To facilitate learning stabilization and clarify the change signal, inspired by DINO-v3, we include additional auxiliary objectives, including a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss, which enhances the geometry of the representation. On the Medical-Diff-VQA benchmark, the model delivers strong BLEU, ROUGE-L, CIDEr, and METEOR scores while offering intrinsic interpretability through the shared saliency mask. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA. Our training strategy also illustrates the potential of a paradigm in utilizing image foundation models in biomedicine: optimizing both supervised and unsupervised learning objectives simultaneously.

Abstract

Medical VQA

Explore similar work

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

InViC: Intent-aware Visual Cues for Medical Visual Question Answering