cs.LGMar 30, 2026
SaveFairboard: a quantitative framework for equity assessment of healthcare models
Organizations: Queen Square Institute of Neurology, University College London, London, UK · Dept. of Physiology, Development and Neuroscience, University of Cambridge, UK · Laboratoire Bordelais de Recherche en Informatique (LABRI), Université de Bordeaux, France
Abstract
Despite there now being more than 1,000 FDA-authorised AI medical devices, formal equity assessments -- whether model performance is uniform across patient subgroups -- are rare. Here, we evaluate the equity of 18 open-source brain tumour segmentation models across 648 glioma patients from two independent datasets (n = 11,664 model inferences) along distinct univariate, Bayesian multivariate, spatial, and representational dimensions. We find that patient identity consistently explains more performance variance than model choice, with clinical factors, including molecular diagnosis, tumour grade, and extent of resection, predicting segmentation accuracy more strongly than model architecture. A voxel-wise spatial meta-analysis identifies neuroanatomically localised biases that are compartment-specific yet often consistent across models. Within a high-dimensional latent space of lesion masks and clinic-demographic features, model performance clusters significantly, indicating that the patient feature space contains axes of algorithmic vulnerability. Although newer models tend toward greater equity, none provide a formal fairness guarantee. Lastly, we release Fairboard, an open-source, no-code dashboard that lowers barriers to equitable model monitoring in medical imaging.
Figures & tables
Fig. 1. Fairboard: an equitable model monitoring dashboard. Schematic overview of the Fairboard platform architecture. Top: Input data types accepted by the dashboard, including tabular performance metrics (individual-level and model-level data) and lesion masks (in zipped NIfTI format). Centre: The Fairboard processing hub, comprising four modules: (i) Univariate Equity —statistics and plotting tools to assess whether model errors are equally distributed across univariate individual factors; (ii) Cohort Equity —multivariable Bayesian and frequentist regressions to identify which patient or model factors predict model errors; (iii) Spatial Equity —brain voxel-wise generalised linear models (GLMs) to test if and where in the brain models are spatially biased; and (iv) Representational Equity —dimensionality reduction (e.g. UMAP) models with latent-space visualisations and GLM evaluations to test whether patterns of model performance cluster by characteristics in a representational latent space. Bottom: Output deliverables include equity metrics, model assessment reports, publication-ready figures, PDF reports, and NIfTI/CSV exports. GLM, generalised linear model; UMAP, uniform manifold approximation and projection; NIfTI, Neuroimaging Informatics Technology Initiative.
Fig. 2. Qualitative segmentation comparison across diverse patient cases. Side-by-side comparison of sample brain tumour segmentations across all dimensions of available diversity in the datasets, illustrating variability in model performance. Patient cases are grouped by clinico-demographic factor: (a–b) molecular diagnosis (Glioblastoma, IDH-wildtype versus Astrocytoma, IDH-mutant), (c–d) age (older versus younger), (e–f) sex (male versus female), (g–h) dataset source (UPENN-GBM versus UCSF-PDGM), and (i–j) neurosurgical resectability (gross total resection versus subtotal resection). For each case, four columns are displayed (left to right): T1-weighted contrast-enhanced MRI scan, ground truth manual segmentation, best-performing model segmentation, and worst-performing model segmentation. Best and worst models were determined by mean DSC averaged across all four tumour compartments. Segmentation overlays: orange = non-enhancing tumour (NET), green = peritumoral oedema (OED), pink = enhancing tumour (ET). DSC, Dice similarity coefficient; GTR, gross total resection; STR, subtotal resection.
Fig. 3. A composite league table of model performance and distributional equity. Compressed league table ranking 18 brain tumour segmentation models across three evaluation frameworks. (a) Performance rank, based on mean segmentation performance across seven metrics (DSC, sensitivity, precision, HD95, NSD@1mm, ASD, volume similarity) and four tumour compartments (WT, NET, ET, OED). (b) Equity rank, based on seven inequality metrics (Gini coefficient, Atkinson index, coefficient of variation, generalised entropy, Hoover index, Theil index, Palma ratio) computed across each of the 28 individual outcome measures (4 compartments 7 metrics) at the patient level. (c) Composite rank under five weighting scenarios ranging from 90% performance / 10% equity to 10% performance / 90% equity. Colour scale: dark red = best rank (1), dark blue = worst rank (18). DSC, Dice similarity coefficient; HD95, 95th-percentile Hausdorff distance; NSD@1mm, normalised surface distance at 1 mm tolerance; ASD, average surface distance; WT, whole tumour; NET, non-enhancing tumour; ET, enhancing tumour; OED, oedema.
Fig. 4. Bayesian linear mixed-effects model of clinico-demographic predictors of segmentation performance. Forest plots showing posterior mean standardised regression coefficients ( ) and 95% highest density intervals (HDI) from Bayesian linear mixed-effects (LME) models with crossed random intercepts for patient ( ) and model ( ). Fixed-effect predictor levels are: Sex [M], Age (Years), Dataset Source [UPENN-GBM], WHO CNS Grade [3], WHO CNS Grade [4], Extent of Resection [STR], Extent of Resection [Biopsy], and WHO 2021 Diagnosis [Non-GBM]. Panels show results for (a) whole tumour (WT), (b) non-enhancing tumour (NET), (c) enhancing tumour (ET), and (d) oedema (OED). Each predictor row displays four colour-coded points representing different performance metrics: teal = DSC, orange = sensitivity, purple = precision, magenta = HD95. Filled circles indicate effects where the 95% HDI excludes zero; open circles indicate non-significant effects. Significance stars are based on the probability of direction (pd): ∗ pd , ∗∗ pd , ∗∗∗ pd . All continuous predictors and dependent variables were z-scored prior to model fitting. M, male; STR, subtotal resection; LME, linear mixed-effects; WT, whole tumour; NET, non-enhancing tumour; ET, enhancing tumour; OED, oedema; HDI, highest density interval; DSC, Dice similarity coefficient; HD95, 95th-percentile Hausdorff distance; pd, probability of direction.
Fig. 5. Spatial equity meta-analysis of voxel-wise segmentation bias. DerSimonian–Laird random-effects meta-analysis of voxel-wise segmentation performance bias across 18 models, Rows correspond to four tumour compartments (WT, NET, ET, OED) and columns correspond to four performance metrics (Dice, sensitivity, precision, HD95). Each cell shows multiple axial brain slices in MNI152 2 mm standard space with voxel-wise combined z-statistics colour-mapped using a diverging colourmap (red/orange = positive bias indicating better performance; blue/cyan = negative bias indicating worse performance; note: for HD95 columns, this interpretation is reversed as higher HD95 values indicate worse boundary accuracy). The DerSimonian–Laird method estimates between-model heterogeneity ( ) and produces a pooled z-statistic at each voxel, accounting for inter-model variability. Voxels are thresholded using Benjamini–Hochberg FDR correction ( ). DSC, Dice similarity coefficient; HD95, 95th-percentile Hausdorff distance; WT, whole tumour; NET, non-enhancing tumour; ET, enhancing tumour; OED, oedema; MNI, Montreal Neurological Institute; FDR, false discovery rate.
Fig. 6. Representational equity analysis of Dice similarity across tumour compartments. High-dimensional representational equity analysis of DSC performance, shown separately for four tumour compartments (rows: WT, NET, ET, OED). Each row contains four panels. Column 1: UMAP scatter plot of individual patients ( ) in a latent space derived from PCA-compressed binary lesion mask features combined with encoded demographic variables. Points are coloured by mean DSC (yellow = high, purple = low). Column 2: Univariate GLM z-score map testing whether the z-scored compartment-specific DSC predicts structure in the UMAP latent space, with Benjamini–Hochberg FDR correction ( ). Column 3: Cohen’s effect sizes comparing significant versus non-significant patient clusters. Column 4: Lesion overlap heatmap in MNI152 standard space showing voxel-wise overlap of lesion masks from significant patients. UMAP, uniform manifold approximation and projection; PCA, principal component analysis; PC, principal component; GLM, generalised linear model; FDR, false discovery rate; MNI, Montreal Neurological Institute; DSC, Dice similarity coefficient; IDH, isocitrate dehydrogenase; WHO, World Health Organization; CNS, central nervous system; M, male; STR, subtotal resection.