eess.IVAug 28, 2026

CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs

Authors: Naren Akash, Arihanth Tadanki, Jayanthi Sivaswamy

Organizations: Center for Visual Information Technology, International Institute of Information Technology Hyderabad, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev.

Figures & tables

Figure 1: A schematic illustration of CheXtriev.
VariantsIRMEdge ConnectivityLocationMLFmAPmHRmRR
V0✕✕✕✕51.839.754.0
V1GTShared Binary✕Global52.939.955.2
V2GTShared Uniform✕Global53.940.856.3
V3GTShared UniformLearnableGlobal54.040.956.4
V4GTUnique Ri−RjR_{i}-R_{j}LearnableLocal51.839.853.7
V5GTUnique Ri−RjR_{i}-R_{j}✕Global54.640.957.1
Table 1: Impact of design choices in CheXtriev on retrieval and ranking performance. Here, IRM and MLF denote inter-anatomic region modelling using graph transformers and multi-level features with gated residuals, respectively.
 Global CNNATH [ 1 ]AnaXNet [ 2 ]CheXtriev
FindingsAPHRRRAPHRRRAPHRRRAPHRRR
LO90.4∗90.4^{\tiny\ast}85.9∗85.9^{\tiny\ast}92.288.2∗88.2^{\tiny\ast}82.2∗82.2^{\tiny\ast}90.988.8∗88.8^{\tiny\ast}82.9∗82.9^{\tiny\ast}91.291.787.693.3
PE68.6∗68.6^{\tiny\ast}54.7∗54.7^{\tiny\ast}72.662.1∗62.1^{\tiny\ast}46.3∗46.3^{\tiny\ast}66.163.7∗63.7^{\tiny\ast}46.2∗46.2^{\tiny\ast}67.271.459.474.7
AT60.3∗60.3^{\tiny\ast}44.1∗44.1^{\tiny\ast}64.254.3∗54.3^{\tiny\ast}36.8∗36.8^{\tiny\ast}57.859.1∗59.1^{\tiny\ast}42.3∗42.3^{\tiny\ast}62.864.449.167.6
ECS64.7∗64.7^{\tiny\ast}48.3∗48.3^{\tiny\ast}67.961.3∗61.3^{\tiny\ast}45.4∗45.4^{\tiny\ast}64.964.7∗64.7^{\tiny\ast}48.5∗48.5^{\tiny\ast}68.469.456.073.7
PE/HO56.5∗56.5^{\tiny\ast}40.2∗40.2^{\tiny\ast}59.251.8∗51.8^{\tiny\ast}34.7∗34.7^{\tiny\ast}54.753.5∗53.5^{\tiny\ast}34.9∗34.9^{\tiny\ast}57.462.547.865.8
Table 2: Comparison of top-5 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)∗(.)^{\ast} indicates p<0.05p<0.05 .
Figure 2: Retrieval performance and saliency map analysis for a sample query image with enlarged cardiac silhouette (ECS). Each row displays the top 4 retrieved images and their corresponding occlusion-based saliency maps generated by CheXtriev, AnaXNet, ATH and Global CNN (top to bottom). A retrieved image is considered correct if it matches the specific finding of interest, in this case ECS, as the query image.
 Global CNNATH [ 1 ]AnaXNet [ 2 ]CheXtriev
FindingsAPHRRRAPHRRRAPHRRRAPHRRR
LO91.4 ∗85.8 ∗91.9 ∗89.7 ∗82.5 ∗90.6 ∗90.7 ∗84.0 ∗91.7 ∗92.588.293.0
PE70.0 ∗56.2 ∗71.262.6 ∗47.6 ∗63.9 ∗64.8 ∗48.6 ∗65.7 ∗72.159.473.0
AT60.8 ∗45.2 ∗61.9 ∗53.9 ∗36.8 ∗54.5 ∗59.7 ∗42.5 ∗61.1 ∗64.948.565.8
ECS64.5 ∗49.3 ∗65.6 ∗61.6 ∗45.9 ∗62.5 ∗64.5 ∗48.3 ∗65.2 ∗70.555.471.7
PE/HO55.3 ∗40.7 ∗56.1 ∗51.0 ∗36.0 ∗51.7 ∗53.7 ∗36.3 ∗54.5 ∗64.049.165.2
Table 3: Comparison of top-3 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)^{{\color[rgb]{0,0,1}\ast}} indicates p<0.05p<0.05 .
 Global CNNATH [ 1 ]AnaXNet [ 2 ]CheXtriev
FindingsAPHRRRAPHRRRAPHRRRAPHRRR
LO89.0 ∗86.1 ∗92.2 ∗86.2 ∗82.0 ∗90.9 ∗87.3 ∗83.5 ∗91.9 ∗90.587.793.2
PE64.4 ∗54.1 ∗73.258.2 ∗45.2 ∗67.2 ∗59.3 ∗46.4 ∗68.9 ∗67.657.575.1
AT55.7 ∗43.5 ∗65.0 ∗50.5 ∗36.1 ∗59.2 ∗55.1 ∗41.4 ∗65.0 ∗59.747.668.6
ECS59.8 ∗48.5 ∗68.5 ∗57.3 ∗43.8 ∗65.7 ∗59.5 ∗46.1 ∗68.3 ∗64.653.174.1
PE/HO52.3 ∗39.1 ∗60.2 ∗48.4 ∗32.8 ∗56.1 ∗50.2 ∗34.0 ∗59.1 ∗59.246.368.0
Table 4: Comparison of top-10 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)^{{\color[rgb]{0,0,1}\ast}} indicates p<0.05p<0.05 .
Radiological Findings#Radiographs
Lung Opacity2679
Pleural Effusion1340
Atelectasis1241
Enlarged Cardiac Silhouette1212
Pulmonary Edema / Hazy Opacity819
Pneumonia643
Table 5: Statistics of the Chest ImageGenome [ 3 ] test dataset, which includes frontal chest radiographs (PA or AP view) with valid bounding box annotations for 18 anatomical regions. The data split is based on the official MIMIC-CXR [ 4 , 5 ] data splits.
More in this topic

Medical Image Retrieval

See all 18 papers

Explore similar work

CardsList
Jul 30, 2026

CXR-Retrieve: Compositional Text-to-Image Retrieval in Chest Radiography

Large chest radiography archives are difficult to search because most studies are paired only with free-text reports rather than structured clinical annotations. Vision-language models offer a natural interface for text-to-image retrieval, but current biomedical models are primarily optimized for report-to-image matching rather than for satisfying short clinical search queries. This creates an objective mismatch: a model may retrieve images related to words in the query while failing to satisfy the full clinical constraint, especially for conjunctions and negations such as ``atelectasis and no pneumonia.'' We introduce CXR-Retrieve, a structured benchmark for compositional chest X-ray text-to-image retrieval. The benchmark contains 5,159 test images from the official test-split of MIMIC-CXR-JPG and 145 textual queries spanning single and conjunction findings, both positive and negative. Relevance is defined by whether a retrieved image satisfies all asserted pathology constraints, rather than by whether it matches a paired report. We further propose a label-aware contrastive fine-tuning objective for clinical retrieval. Our method attracts image-text pairs with compatible asserted pathology constraints, including shared confirmed absences, while explicitly repelling contradictory pairs. Starting from the in-domain CXR-CLIP checkpoint, our method improves Precision@5 over CXR-CLIP by 8.5 percentage points on two-pathology conjunctions and by 22.0 percentage points on negation queries. These results show that reliable chest X-ray retrieval requires training objectives that model not only which findings are mentioned, but also how they are clinically asserted.

Tomer Erez, Moshe Kimhi, Chaim Baskin +1

Technion – Israel Institute of Technology · Ben-Gurion University of the Negev

Contrastive LearningMedical ImagingImage-Text RetrievalMedical Image BenchmarksMedical Image Retrieval
AskPDF
May 17, 2026

RadGenome-Anatomy: A Large-Scale Anatomy-Labeled Chest Radiograph Dataset via Physically Grounded Volumetric Projection

Anatomical structure labels for chest radiographs are essential for medical image segmentation and a broad range of downstream diagnostic tasks. However, annotating anatomy directly on 2D chest radiographs is labor-intensive and intrinsically ambiguous, as 3D anatomical structures are projected onto a single 2D plane where boundaries may overlap, be occluded, or appear only partially visible. Consequently, existing anatomy-labeled chest radiograph datasets remain limited in scale, anatomy coverage, and label reliability. To address these limitations, we introduce RadGenome-Anatomy, the largest anatomy-labeled chest radiograph dataset, containing over 10 million segmentation masks across 210 anatomical structures in 25,692 studies. It is constructed by projecting large-scale 3D anatomical masks from CT volumes into 2D radiographic space through canonical radiographic geometry. This shifts annotation from directly tracing uncertain 2D boundaries to defining anatomy in volumetric space, where structures that overlap or become partially invisible in radiographs remain spatially separable. As a result, each 2D mask represents the physically grounded projected footprint of a volumetrically defined structure. The scale and broad anatomical coverage of RadGenome-Anatomy, including structures that are overlapping, partially visible, or difficult to delineate directly, enable research on geometric measurements as explicit evidence for chest radiograph interpretation. We demonstrate this by training XAnatomy to predict structure-specific masks and derive clinically relevant measurements, achieving diagnostic accuracies of 96.4%, 95.6%, and 89.2% for cardiomegaly, kyphosis, and scoliosis, respectively.

Shuchang Ye, Mingyuan Meng, Hao Wang +2

The University of Sydney · Zhongguancun Academy · Shanghai Jiao Tong University +1

Medical ImagingMedical Image SegmentationMedical Image Benchmarks
AskPDF
Jun 7, 2026

CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs

Vision-language models (VLMs) pretrained on large-scale image-text pairs demonstrate strong image-level understanding, but are primarily optimized for global alignment and do not explicitly encode fine-grained anatomical structure, limiting their suitability for spatially precise tasks such as segmentation. We introduce CheXanatomy, a framework that integrates explicit anatomical knowledge into a pretrained VLM through autoregressive token-space supervision. Instead of adding task-specific decoder heads, the model is trained to generate anatomical segmentation masks via next-token prediction. To enable scalable supervision, we synthesize realistic chest radiographs from CT volumes and forward-project CT segmentation labels to obtain anatomically consistent 2D masks. We evaluate the approach on synthetic and real chest radiographs against a U-Net baseline, including ablations on model scale, input resolution, and vision encoder fine-tuning. Autoregressive anatomical supervision achieves performance comparable to specialized convolutional models in-distribution and demonstrates improved geometric robustness under domain shift to real CXR data. In addition, anatomy-pretrained models exhibit improved sample efficiency when adapting to novel localization tasks under limited supervision. Larger models and higher input image resolution improve performance, while vision encoder fine-tuning has limited effect. These results show that embedding anatomical structure directly into the generative objective promotes spatially grounded representations and supports anatomy-aware medical vision-language modeling.

Sergios Gatidis, Curtis Langlotz, Christian Bluethgen

Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA

Radiology VLMsMedical Image SegmentationMedical Image Representation Learning
AskPDF