Evaluating the Impact of Medical Image Reconstruction on Downstream AI Fairness and Performance
Authors: Matteo Wohlrapp, Niklas Bubeck, Daniel Rueckert, William Lotter
Organizations: Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA · AI in Medicine, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Department of Computing, Imperial College London, London, UK · Harvard Medical School, Boston, MA, USA
AI-based image reconstruction models are increasingly deployed in clinical workflows to improve image quality from noisy data, such as low-dose X-rays or accelerated MRI scans. However, these models are typically evaluated using pixel-level metrics like PSNR, leaving their impact on downstream diagnostic performance and fairness unclear. We introduce a scalable evaluation framework that applies reconstruction and diagnostic AI models in tandem, which we apply to two tasks (classification, segmentation), three reconstruction approaches (U-Net, GAN, diffusion), and two data types (X-ray, MRI) to assess the potential downstream implications of reconstruction. We find that conventional reconstruction metrics poorly track task performance, where diagnostic accuracy remains largely stable even as reconstruction PSNR declines with increasing image noise. Fairness metrics exhibit greater variability, with reconstruction sometimes amplifying demographic biases, particularly regarding patient sex. However, the overall magnitude of this additional bias is modest compared to the inherent biases already present in diagnostic models. To explore potential bias mitigation, we adapt two strategies from classification literature to the reconstruction setting, but observe limited efficacy. Overall, our findings emphasize the importance of holistic performance and fairness assessments throughout the entire medical imaging workflow, especially as generative reconstruction models are increasingly deployed.
Figures & tables
Figure 1: Combined pipeline for downstream bias evaluation and mitigation in medical image reconstruction. MRI and X-ray images undergo realistic simulated degradation and are subsequently reconstructed with three approaches before serving as input to downstream prediction models. Reconstruction quality, downstream performance, and fairness are evaluated. Subsequently, two bias mitigation strategies are applied exclusively during reconstruction fine-tuning.
Figure 2: Downstream performance and PSNR at varying noise levels. Axes for PSNR and task performance are scaled to comparable percentage ranges. Although PSNR declines as noise increases, task performance remains stable. Baseline indicates performance on original images.
Figure 3: Distribution of bias changes (percent change compared to original images) across all reconstruction models, datasets, and tasks, stratified by sensitive attribute. The vertical lines mark the medians. Most shifts cluster near zero, but sex shows a broader positive tail. Separate plots by sensitive attribute are contained in Fig. 20 .
Figure 4: Equalized odds bias change pre- and post-mitigation compared to predictions on original images for CheXpert. Pre-mitigation (“Reconstruction”), bias tends to increase slightly for sex; race exhibits high variance. Bias tends to decline slightly post-mitigation. Error bars represent standard deviation.
Sex
Age
Race
Classification
0.05±0.07
0.17±0.08
0.07±0.03
Segmentation
1.10
1.22
–
Table 1: Baseline fairness of the classifiers (EODD) and the segmentation model (SER) for different sensitive attributes. For classification, mean and s.d. are reported across all classification tasks. Segmentation corresponds to UCSF-PDGM performance. Sex exhibits the lowest baseline bias.
U-Net
GAN
SDE
Median
2.28
-0.21
1.59
Absolute Median
14.6
11.8
11.5
Table 2: Median of bias change (% change in EODD/SER) by reconstruction approach across all datasets, tasks, and attributes by model. SDE and GAN show a slightly smaller bias shift than U-Net.
Sex
Age
Race
Standard
24.1
-1.88
3.30
Reweighted
10.6
0.03
1.05
EODD
7.56
-2.01
0.52
Table 3: Median bias change (% change in EODD/SER) by mitigation strategy across all datasets, tasks, and models. Standard corresponds to the original results without mitigation applied.
Reweighted
EODD
Chex
UCSF
Chex
UCSF
PSNR
0.54
-0.75
-0.64
-7.28
Down.
0.07
-1.97
0.02
-2.94
Table 4: Mean change (%) in PSNR and downstream performance (AUROC/Dice) per dataset after each mitigation averaged over reconstruction models and tasks. Performance drops are modest, except for PSNR in UCSF-PDGM for EODD.
Figure 5: EODD and SER bias change pre- and post-mitigation compared to predictions on original images for UCSF-PDGM tasks. No consistent trends emerge for the classification tasks. Error bars represent standard deviation.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: X-Ray images with photon count 100,000, 10,000, 3,000.
Figure 7: MRI images with acceleration 4, 8, 16.
Figure 8: Reconstruction example from photon count 10,000 for the different models. Grad-CAM Selvaraju et al. (2017) and logit score correspond to the lung lesion prediction of the pre-trained classifier, indicating similar predictions on the reconstructed images.
Figure 9: Reconstruction with corresponding segmentation and Dice score of an MRI image with acceleration 8 for the different models.
AI/AN
Asian
Black
NH/PI
Other
White
Female, >62
54
1539
923
314
2518
6456
11804
Female, ≤62
39
1739
608
136
1710
9500
13732
Male, >62
56
1734
1023
240
3553
8984
15590
Male, ≤62
27
1924
539
171
1853
11170
15684
176
6936
3093
861
9634
36110
56810
Appendix
Table 5: Patient-wise groups used for analysis based on sex, age, and race for the CheXpert dataset. Unequally distributed with very few samples for American Indian or Alaska Native (AI/AN) and Native Hawaiian or Other Pacific Islander (NH/PI).
Male
Female
≤58
155
92
147
>58
144
110
254
299
202
501
Appendix
Table 6: Patient distribution by sex and age for the UCSF-PDGM dataset. Patients under 58 and females represent minority groups.
Figure 10: Tumor Type and Tumor Grade and PSNR values for different noise levels on UCSF-PDGM. The image quality and diagnostic performance axes are on a similar percentage scale. Task performance metrics show high stability across models and noise conditions, while PSNR drops with increasing noise.
Photon Count
Metrics
Baseline
U-Net
GAN
SDE
100,000
AUROC
Atalectasis
0.87
0.87
0.86
0.87
Cardiomegaly
0.91
0.91
0.91
0.91
Consolidation
0.91
0.91
0.91
0.91
Edema
0.90
0.90
0.90
0.90
EC
0.79
0.78
0.78
0.79
Fracture
0.76
0.75
0.75
0.76
Appendix
Table 7: CheXpert performance across reconstruction models and photon counts. Pathologies with lower baseline AUROC (e.g., fracture, pneumothorax, lung lesion) experience greater performance drops under noise compared to more easily detectable conditions (e.g., effusion, cardiomegaly). Baseline corresponds to original images.
Acceleration
Metrics
Baseline
U-Net
GAN
SDE
4
AUROC
Tumor Type
0.79
0.79
0.77
0.78
Tumor Grade
0.73
0.76
0.71
0.72
Dice
0.72
0.72
0.71
0.72
PSNR
42.94
37.71
40.23
LPIPS
0.01
0.02
0.00
8
AUROC
Tumor Type
0.79
0.77
0.83
0.79
Appendix
Table 8: Performance metrics for UCSF-PDGM across reconstruction models and noise levels. While PSNR varies with noise and model, downstream segmentation and classification metrics remain relatively stable, indicating robust task performance across conditions.
Figure 11: Influence of fairness weighting parameter ( λfair ) on classifier AUROC performance and fairness metrics for the Equalized Odds (EODD) mitigation constraint, evaluated with U-Net on the CheXpert dataset. There is minor sensitivity of AUROC to lambda; fairness metrics show greater variance but minimal substantial improvement with increased λ .
Figure 12: Impact of λfair on reconstruction quality (PSNR) compared to fairness for the EODD constraint mitigation. PSNR remains stable across lambda variations, while fairness shows slight variation without substantial improvement.
Figure 13: Change in prediction performance after applying bias mitigation techniques. Each row compares two datasets for a given method: (a) Reweighted sampling, (b) Equalized odds constraint. UCSF-PDGM experiences more performance degradation. However, all techniques show good stability in task performance, with few outliers in the UCSF-PDGM dataset.
Figure 14: Equality of opportunity (EOP) bias change pre- and post-mitigation compared to predictions on original images for CheXpert classification. Pre-mitigation, bias tends to increase slightly for sex; race exhibits high variance. Bias tends to decline slightly post-mitigation.
Figure 15: Equality of opportunity (EOP) and Δ Dice bias change compared to predictions on original images pre- and post-mitigation for UCSF-PDGM classification and segmentation.
Figure 16: Distribution of bias changes when using alternative race subgroups for CheXpert calculations.
Figure 17: Equalized odds bias change pre- and post-mitigation compared to predictions on original images for CheXpert when using alternative race subgroups.
Figure 18: Downstream performance and SSIM at varying noise levels. Axes for SSIM and task performance are scaled to comparable percentage ranges. Baseline indicates performance on original images.
Figure 19: Downstream performance and SSIM at varying noise levels on classification tasks in UCSF-PDGM. Axes for SSIM and task performance are scaled to comparable percentage ranges. Baseline indicates performance on original images.
Figure 20: Distribution of bias changes separated by sensitive attribute.
Medical imaging is central to modern diagnostics, and artificial intelligence (AI) systems are increasingly used to support image-based analysis by improving efficiency, accuracy, and access to care. However, inequities in healthcare access and differential disease prevalence create severe demographic imbalances in clinical image data. Such imbalances are compounded by the fact that diseases can manifest with distinct features across demographic groups, rendering certain phenotypic presentations naturally rare. AI models trained on such imbalanced data risk perpetuating diagnostic bias and widening healthcare disparities. Here we introduce FairGen, a fairness-aware diffusion framework that synthesizes demographically balanced medical images while preserving pathology-relevant visual features. By embedding physician-aligned preferences into the generation process, FairGen improves subgroup coverage during synthesis and downstream classification. Applied to dermatology, radiology, and neuroimaging benchmark tasks, FairGen achieves fairness improvements of 95.9% for skin images, 80.0% for chest radiography, and 35.2% for brain MRI, while maintaining competitive diagnostic accuracy relative to models trained on original clinical data. Clinician-facing expert review and external validation on independent cohorts further support that these gains extend beyond standard fidelity metrics and are not confined to the original in-distribution datasets.
Zhimin Li, Ruichen Zhang, Zhen Tan +3
University of Pittsburgh, Swanson School of Engineering, Pittsburgh, 15261, USA · 2The University of North Carolina at Chapel Hill, Department of Computer Science, Chapel Hill, 27599, USA · 3Arizona State University, School of Computing and Augmented Intelligence, Tempe, 85281, USA +1
Medical image generators trained on imbalanced data can fail at demographic intersections absent from training. We introduce CompDiff, which encodes age, sex and race separately and composes supervised demographic tokens alongside clinical text. Across chest radiographs and fundus images, CompDiff improves overall and subgroup fidelity relative to prompt conditioning (RoentGen-v2) and loss reweighting (FairDiffusion). It generalises in zero-shot generation to 16 chest X-ray intersections excluded from training, achieving the lowest mean FID-RadImageNet in every intersection. In a blinded reader study of these unseen intersections, two radiologists gave CompDiff the highest mean scores among generators for anatomical realism and agreement with the clinical impression, and selected its images most often as the most realistic. Pretraining with CompDiff images improved downstream classification, while CompDiff audit cohorts reduced estimation error on rare intersections. These findings support compositional demographic conditioning for extending medical image synthesis to underserved populations. Code: https://github.com/mahmoudibrahim98/CompDiff
Mahmoud K. Ibrahim, Bart Elen, Chang Sun +5
Institute of Data Science, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · VITO, Belgium
Fairness audits of medical imaging models commonly report the largest performance difference between demographic subgroups and treat any positive value as evidence of bias. Yet a perfectly fair model also produces a positive difference, which grows as subgroups shrink. Some mitigation methods remove demographic information from models because they assume that it produces the difference. Here we introduce Fair-model Reference And Mechanism Evaluation (FRAME). Its first step compares each reported difference with the difference expected from a perfectly fair model at the same subgroup counts. Its second step tests whether candidate causes of any excess over this fair-model reference change the difference when they are injected into model features. We evaluated 36 encoders on nine datasets in three modalities and audited 89 published differences across six modalities. For 10 frozen encoders and 13 chest radiograph findings, the reference is a median 41% of the race difference and 22% of the age difference. It is exceeded by 40 of 53 published differences in sensitivity or false positive rate and by one of 36 in the area under the receiver operating characteristic curve (AUROC). Adding race information to the features of two encoders does not change the race difference significantly. At matched disease performance, mitigation reduces the race difference by less than its spread across three pretraining seeds (medians 0.005 and 0.012) and the age difference by more (0.008 and 0.006). Image-text pretraining raises the worst-group AUROC for race by a median 0.049 over self-supervised pretraining. Reporting the fair-model reference beside every new and published subgroup difference could direct mitigation to the differences above it.
Mahshad Lotfinia, Daniel Truhn, Andreas Maier +1
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany