LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray

Myeongkyun Kang; Yanting Yang; Xiaoxiao Li

cs.CVMar 19, 2026

LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray

Authors: Myeongkyun Kang, Yanting Yang, Xiaoxiao Li

Organizations: The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada

Abstract

Fine-grained representation learning is crucial for retrieval and phrase grounding in chest X-rays, where clinically relevant findings are often spatially confined. However, the lack of region-level supervision in contrastive models and the limited ability of large vision language models to capture fine-grained representations in external validation lead to suboptimal performance on these tasks. To address these limitations, we propose Location-aware Fine-grained representation learning (LoFi), which jointly optimizes sigmoid, captioning, and location-aware captioning losses using a lightweight large language model. The location-aware captioning loss enables region-level supervision through grounding and dense captioning objectives, thereby facilitating fine-grained representation learning. Building upon these representations, we integrate a fine-grained encoder into retrieval-based in-context learning to enhance chest X-ray grounding across diverse settings. Extensive experiments demonstrate that our method achieves superior retrieval and phrase grounding performance on MIMIC-CXR and PadChest-GR.

Figures & tables

Figure 1: Illustration of (a) location-aware fine-grained representation learning frameworks and (b) a retrieval-based in-context learning pipeline.

Model	Image-to-Text					Text-to-Image
Model	R@1	R@5	R@10	R@20	R@40	R@1	R@5	R@10	R@20	R@40
SigLIP2 [ 21 ]	0.08	0.95	1.56	2.59	4.32	0.27	0.98	1.60	2.95	4.69
BiomedCLIP [ 27 ]	0.82	2.10	3.78	7.07	12.66	0.58	1.77	3.17	6.00	11.02
BMC-CLIP [ 14 ]	0.12	0.53	1.15	2.55	4.44	0.05	0.80	1.65	2.63	4.78
MedSigLIP [ 18 ]	4.61	14.27	21.92	32.69	44.82	2.38	10.33	17.47	26.43	38.48
CARZero [ 11 ]	11.64	29.52	40.17	51.19	63.98	9.67	26.02	36.24	48.40	62.37

Table 1: Retrieval performance on the MIMIC-CXR dataset. R@K denotes Recall at K, where K is the cutoff rank.

Model	$k$ =20					$k$ =40
Model	Ro/L	Ro/S	[email protected]	[email protected]	[email protected]	Ro/L	Ro/S	[email protected]	[email protected]	[email protected]
SigLIP2 [ 21 ]	52.50	42.14	17.75	18.61	16.96	56.87	44.70	21.47	22.53	20.51
BiomedCLIP [ 27 ]	56.19	42.39	18.43	19.57	17.42	60.14	45.40	22.06	22.94	21.24
BMC-CLIP [ 14 ]	53.24	42.20	17.15	18.16	16.24	56.71	44.66	20.92	21.95	19.99
MedSigLIP [ 18 ]	58.79	44.69	21.77	22.62	20.97	61.63	46.89	24.51	25.12	23.93
RadIR [ 28 ]	56.03	42.13	17.50	18.54	16.57	56.34	41.72	17.04	17.77	16.37

Table 2: Phrase grounding performance on the PadChest-GR dataset for external validation using retrieval-based ICL with MedGemma and

k

demonstrations.

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 260 papers

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models
Aug 2, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Imaging Foundation Models Medical Image Grounding
The University of British Columbia · Vector Institute
CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation
Aug 31, 2026Adonay Demewez Gebremedhin, Wessam Shehieb, Sara Alansari +4Radiology Report Generation Radiology VLMs
Ajman University Ajman, United Arab Emirates · University of Birmingham Birmingham, United Kingdom · Khalifa University Abu Dhabi, United Arab Emirates +1
Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Tasks
Sep 25, 2026Denis Musinguzi, Andrew Katumba, Prasenjit MitraVision-Language Pretraining Radiology VLMs
Carnegie Mellon University · Makerere University

Aug 2, 2026J/K move · Enter open · S save

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

Myeongkyun Kang, Yanting Yang, Xiaoxiao Li

The University of British Columbia · Vector Institute

Medical Imaging Foundation Models Medical Image Grounding Visual Grounding Medical Image Representation Learning Fine-Grained Visual Perception

Open paper AskPDF arXiv

Fine-grained visual representations are essential for medical image analysis, particularly when diagnostically relevant evidence is subtle and spatially localized. Modern transformer-based medical vision encoders must therefore learn patch-level representations that are both clinically meaningful and spatially consistent. Without these properties, large vision-language models (LVLMs) operate on an ambiguous visual foundation, limiting their ability to generate clinically reliable and spatially grounded responses. However, existing training strategies for medical vision encoders rarely achieve both objectives. Image-text alignment provides clinically meaningful supervision primarily at the image level, leaving the spatial localization of diagnostic evidence weakly constrained. In contrast, self-supervised learning promotes spatial consistency but lacks the semantic supervision needed to distinguish visually similar yet clinically distinct regions. To address this gap, we present LoFi, a medical vision foundation model built on location-aware fine-grained representation learning. LoFi trains a vision encoder with a lightweight large language model under grounding and grounded captioning objectives. Because these objectives require predicting location from clinical text and vice versa, spatial consistency emerges without any explicit patch-level regularization. To enable training at scale, we construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets curated from 84 datasets spanning 7 modalities. Across phrase grounding, visual question answering, and region-based organ classification under perturbations, LoFi consistently outperforms general-purpose and medical vision foundation models as well as state-of-the-art LVLMs. Code is available at https://github.com/myeongkyunkang/lofi-medg.

Abstract

Medical Image Representation Learning

Explore similar work

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation

Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Tasks