AnyCXR: Human Anatomy Segmentation of Chest X-ray at Any Acquisition Position using Multi-stage Domain Randomized Synthetic Data with Imperfect Annotations and Conditional Joint Annotation Regularization Learning
Organizations: Department of Radiology, Northwestern University, Chicago, IL, USA · Data Science Institute, Vanderbilt University, Nashville, TN, USA · Department of Orthopedics, The Second Hospital of Shanxi Medical University, Taiyuan, Shanxi, P.R. China · Second Clinical Medical College, Shanxi Medical University, Taiyuan, Shanxi, P.R. China · School of Computer Science, University of Sydney, Sydney, NSW, Australia
Robust anatomical segmentation of chest X-rays (CXRs) remains challenging due to the scarcity of comprehensive annotations and the substantial variability of real-world acquisition conditions. We propose AnyCXR, a unified framework that enables generalizable multi-organ segmentation across arbitrary CXR projection angles using only synthetic supervision. The method combines a Multi-stage Domain Randomization (MSDR) engine, which generates over 100,000 anatomically faithful and highly diverse synthetic radiographs from 3D CT volumes, with a Conditional Joint Annotation Regularization (CAR) learning strategy that leverages partial and imperfect labels by enforcing anatomical consistency in a latent space. Trained entirely on synthetic data, AnyCXR achieves strong zero-shot generalization on multiple real-world datasets, providing accurate delineation of 54 anatomical structures in PA, lateral, and oblique views. The resulting segmentation maps support downstream clinical tasks, including automated cardiothoracic ratio estimation, spine curvature assessment, and disease classification, where the incorporation of anatomical priors improves diagnostic performance. These results demonstrate that AnyCXR establishes a scalable and reliable foundation for anatomy-aware CXR analysis and offers a practical pathway toward reducing annotation burdens while improving robustness across diverse imaging conditions.
Figures & tables
Figure 1: AnyCXR Data Pipeline: This workflow projects 3D CT volumes into diverse 2D Digitally Reconstructed Radiographs (DRRs) and segmentation masks using Multi-stage Domain Randomization. The pipeline includes an automated filtering step that discards invalid masks (red). The feature plot (bottom left) demonstrates that this method generates a data distribution significantly broader than typical real-world datasets.
Figure 2: MSDR Pipeline: This flow chart details the Multi-stage Domain Randomization engine, which converts 3D CT data into 2D synthetic DRRs through a sequence of stochastic steps. The pipeline involves Pre-projection Randomizations acting on Hounsfield Units and image artifacts, In-projection Randomizations modifying geometry, and Post-projection Randomizations simulating detector response and inverting polarity. The red boxes denote randomization names, while the blue boxes denote different approaches.
Figure 4: DRR Augmentation Demonstration for Major Innovation: This figure visually illustrates the effect of the primary Multi-stage randomization on the resulting DRR, starting from a Default DRR (left panel). Each subsequent column shows a different major 3D randomization applied in isolation: Bone-to-soft-tissue, Inter-bone, Intra-bone, Soft-tissue, and Projection geometry randomization, demonstrating how these 3D perturbations introduce diverse image appearances and contrast variations.
Figure 5: CAR Framework Overview: Input X-rays are segmented by a U-Net, with predictions substituting missing ground truth labels. Conditioned on image X , these maps are encoded by Gθ1 to align their latent distributions via LDist (optimizing the U-Net), while the decoder Gθ2 ensures latent integrity via the reconstruction loss LRecon .
Figure 6: Segmentation Comparison on Synthetic DRR: This figure presents a side-by-side comparison of ground truth and model Prediction for major anatomical groups on the TotalSegmentator dataset. White circles indicate specific regions where the prediction deviated from the ground truth. While overall organ localization is highly accurate, the primary challenge is focused on subtle boundary adherence and fine detail resolution. The segmentation errors, particularly the fine organ boundaries of lungs and hearts, stem from the inherent difficulty in precisely delineating edges in the 2D projection plane, where overlap is severe and the model must rely heavily on learned anatomical priors to infer invisible or faint structures.
Figure 7: Detailed Comparison of Segmentation on Real-World CXRs and Expert Refinement: This figure presents a comparison of the model’s prediction against the expert corrections on real PA and LA view radiographs, illustrating the editing required to obtain clinically acceptable masks. White circles indicate specific regions where the prediction deviated from the expert annotation. While AnyCXR provides a highly effective initial segmentation proposal, these differences reflect areas of necessary expert editing. Specific challenges arise from image contrast variations and overexposure in clinical data, making detailed editing difficult, particularly for fine structures like the ribs and lungs. Furthermore, a non-trivial portion of the divergence relates to the inherent clinical ambiguity of 2D projection, such as precise lung lobe placement and boundary definition in superimposed regions, where the expert’s final delineation corrects the prediction based on clinical consensus.
View
Method
CAR
Overall ↑
Heart & Vessels ↑
Lung ↑
Rib ↑
Vertebrae ↑
Other Bone ↑
PA
DenseUNet
Off
0.825
0.861
0.890
0.752
0.795
0.829
PA
DenseUNet
On
0.844
0.872
0.900
0.783
0.825
0.843
PA
SegFormer
Off
0.841
0.875
0.916
0.742
0.821
0.852
PA
SegFormer
On
0.854
0.882
0.921
0.768
0.836
0.865
PA
Rad-DINO
Off
0.857
0.897
0.930
0.742
0.862
0.854
PA
Rad-DINO
On
0.863
0.892
0.934
0.747
0.874
0.869
Table 4: DenseUNet, SegFormer, and Rad-DINO performance of DSC with and without CAR
Figure 8: Ablation Study of the CAR model: This chart compares the DSC performance of the Full Aug model (MSDR + PostHoc augmentations) against the final CAR + Aug framework. The CAR framework consistently improves performance across all groups, with the largest gain ( +1.51% ) seen in the challenging Lateral Ribs group, validating CAR’s efficacy in enforcing anatomical priors where augmentation alone is insufficient.
Figure 9: Real CXR Segmentation Performance Comparison: This chart compares the DSC performance of FluoroSAM against AnyCXR on expert-refined real CXR annotations across PA and LA views. AnyCXR achieves substantially higher DSC scores across all evaluated organ groups, demonstrating stronger generalization to real CXR segmentation.
Figure 10: Segmentation comparison of FluoroSAM and AnyCXR on CXR images in PA and LA views.
Figure 11: Scaling Synthetic Supervision: This figure demonstrates the DSC of Full MSDR with and without CAR across different configurations. The 500,000 and 1,000,000 DRR configurations are generated under relaxed QC thresholds.
Figure 12: Visualization for CTR disease detection: This figure demonstrates the utility of the segmentation model for automatic Cardiothoracic Ratio (CTR) assessment in the PA view. The left panel shows the delineated heart and the measurement lines used to calculate the CTR score, distinguishing between a normal case and a pathological case. The right panels illustrate the ROC curves and distribution of derived CTR scores, achieving an overall AUC of 0.93 with statistically significant separation between normal and cardiomegaly subjects across age and gender groups.
Figure 13: Visualization for SPCA disease detection: This figure demonstrates the clinical utility of instance-level vertebral segmentation for preliminary Spine Curvature Assessment (SPCA). The left panel illustrates the identification and tracking of thoracic vertebral centroids to measure deviation, distinguishing between a normal spine and a case with probable scoliosis. The center and right panels show the distribution of SPCA scores and the ROC curve, achieving an AUC of 0.87, which validates its effectiveness as a screening tool for spinal deformities.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Vision-language models (VLMs) pretrained on large-scale image-text pairs demonstrate strong image-level understanding, but are primarily optimized for global alignment and do not explicitly encode fine-grained anatomical structure, limiting their suitability for spatially precise tasks such as segmentation. We introduce CheXanatomy, a framework that integrates explicit anatomical knowledge into a pretrained VLM through autoregressive token-space supervision. Instead of adding task-specific decoder heads, the model is trained to generate anatomical segmentation masks via next-token prediction. To enable scalable supervision, we synthesize realistic chest radiographs from CT volumes and forward-project CT segmentation labels to obtain anatomically consistent 2D masks. We evaluate the approach on synthetic and real chest radiographs against a U-Net baseline, including ablations on model scale, input resolution, and vision encoder fine-tuning. Autoregressive anatomical supervision achieves performance comparable to specialized convolutional models in-distribution and demonstrates improved geometric robustness under domain shift to real CXR data. In addition, anatomy-pretrained models exhibit improved sample efficiency when adapting to novel localization tasks under limited supervision. Larger models and higher input image resolution improve performance, while vision encoder fine-tuning has limited effect. These results show that embedding anatomical structure directly into the generative objective promotes spatially grounded representations and supports anatomy-aware medical vision-language modeling.
Sergios Gatidis, Curtis Langlotz, Christian Bluethgen
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA
Deep learning models for chest X-ray diagnosis are constrained by limited coverage of clinically meaningful concept combinations in publicly available training datasets. While synthetic image generation has been explored to increase data diversity, existing methods rarely enforce clinical or anatomical constraints, limiting utility for improving model reliability. We propose CARPA, a clinically aware and anatomically grounded framework for synthetic chest X-ray generation that applies targeted perturbations to clinical concept vectors while preserving anatomical structure. By producing anatomically faithful synthetic images with controlled concept insertions and deletions, CARPA expands clinically relevant concept coverage. We evaluate CARPA across seven backbone architectures by fine-tuning models on synthetic subsets and testing on a held-out MIMIC-CXR benchmark. Compared to prior concept perturbation approaches, fine-tuning on CARPA-generated images consistently improves precision-recall performance, reduces predictive uncertainty, and improves model calibration. Structural and semantic analyses demonstrate high anatomical fidelity, strong concept alignment, and low semantic uncertainty. Evaluation by two expert radiologists further confirms realism and clinical agreement. Together, these results show that anatomically grounded concept perturbations enable more effective use of synthetic data, improving both performance and reliability of chest X-ray classification models and supporting safer clinical deployment.
Anatomical structure labels for chest radiographs are essential for medical image segmentation and a broad range of downstream diagnostic tasks. However, annotating anatomy directly on 2D chest radiographs is labor-intensive and intrinsically ambiguous, as 3D anatomical structures are projected onto a single 2D plane where boundaries may overlap, be occluded, or appear only partially visible. Consequently, existing anatomy-labeled chest radiograph datasets remain limited in scale, anatomy coverage, and label reliability. To address these limitations, we introduce RadGenome-Anatomy, the largest anatomy-labeled chest radiograph dataset, containing over 10 million segmentation masks across 210 anatomical structures in 25,692 studies. It is constructed by projecting large-scale 3D anatomical masks from CT volumes into 2D radiographic space through canonical radiographic geometry. This shifts annotation from directly tracing uncertain 2D boundaries to defining anatomy in volumetric space, where structures that overlap or become partially invisible in radiographs remain spatially separable. As a result, each 2D mask represents the physically grounded projected footprint of a volumetrically defined structure. The scale and broad anatomical coverage of RadGenome-Anatomy, including structures that are overlapping, partially visible, or difficult to delineate directly, enable research on geometric measurements as explicit evidence for chest radiograph interpretation. We demonstrate this by training XAnatomy to predict structure-specific masks and derive clinically relevant measurements, achieving diagnostic accuracies of 96.4%, 95.6%, and 89.2% for cardiomegaly, kyphosis, and scoliosis, respectively.
Shuchang Ye, Mingyuan Meng, Hao Wang +2
The University of Sydney · Zhongguancun Academy · Shanghai Jiao Tong University +1