BatchBench: Benchmarking AI Robustness to Batch Effects and Domain Shift in Biomedical Images - A JUMP Cell Painting Leaderboard

Sonja Topf; Ana Sanchez-Fernandez; Günter Klambauer

cs.LGOct 21, 2026

BatchBench: Benchmarking AI Robustness to Batch Effects and Domain Shift in Biomedical Images - A JUMP Cell Painting Leaderboard

Authors: Sonja Topf, Ana Sanchez-Fernandez, Günter Klambauer

Organizations: ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University, Linz, Austria · Clinical Research Institute for Medical AI, Johannes Kepler University, Linz, Austria

Abstract

Despite recent advances, current AI systems remain highly susceptible to domain shifts, particularly in biomedical imaging, where images are usually acquired under substantially different experimental conditions. Although recent domain adaptation methods have made substantial progress, significant performance gaps under domain shift persist. To measure progress in this regard, we introduce BatchBench, a leaderboard that evaluates and compares the robustness of AI systems on biomedical images from unseen experimental batches and data sources. BatchBench is built on JUMP-CP, one of the largest publicly available biomedical imaging datasets. Generated using the Cell Painting assay, JUMP-CP exhibits pronounced batch effects and domain shifts, making it a challenging testbed for evaluating robustness to distributional changes.

Figures & tables

Figure 1: Source-level split of the BatchBench leaderboard. Nine sources of JUMP-CP-2k are public (eight for training, source 2 for validation). Sources 4, 6, 9, and 13 are held out as the private test set, so that every leaderboard entry is evaluated on unseen data-generating sources.

Split	Subset	Compounds	Samples	Samples/compound	Sources	Batches	Plates
Public (train + val)	POSCON	8	48,574	6,071.75	9	102	1,534
	TARGET	293	23,455	80.05	7	63	76
	REST	1,926	23,186	12.04	9	85	931
Private test	POSCON	8	16,691	2,086.38	4	42	370
	TARGET	293	25,352	86.53	4	40	62
	REST	1,864	4,107	2.20	2	18	179

Table 1: Statistics of JUMP-CP-2k per split and evaluation subset. The private test set of the leaderboard consists of the held-out sources 4, 6, 9, and 13. Samples/compound is the mean number of images per compound. Source, batch, and plate identifiers are provided for every sample.

Entry	Overall	POSCON	TARGET	REST
ResNet18 (no adaptation)	$0.175\pm 0.015$	$0.465\pm 0.036$	$0.012\pm 0.004$	$0.002\pm 0.000$
DANN	$0.170\pm 0.011$	$0.465\pm 0.026$	$0.003\pm 0.004$	$0.001\pm 0.001$
deepCORAL	$0.201\pm 0.006$	$0.502\pm 0.024$	$0.034\pm 0.006$	$0.007\pm 0.002$
AdaBN	$0.244\pm 0.005$	$0.646\pm 0.005$	$0.019\pm 0.005$	$0.002\pm 0.001$
TENT	$0.249\pm 0.007$	$0.661\pm 0.011$	$0.017\pm 0.005$	$0.002\pm 0.001$
CS-ARM-BN	$0.256\pm 0.004$	$0.660\pm 0.005$	$0.031\pm 0.004$	$0.004\pm 0.001$

Table 2: Baseline entries of the BatchBench leaderboard, evaluated on the private test set (held-out sources 4, 6, 9, and 13). Values are the mean

\pm

sample standard deviation over three runs with different random seeds. Chance level is

1/2{,}000=0.0005

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 279 papers

Stabilizing In-Context Multi-Source Domain Adaptation for Biomedical Images Through Controls
Apr 22, 2026Ana Sanchez-Fernandez, Thomas Pinetz, Werner Zellinger +1Multi-Source Domain Adaptation Batch Effect Correction
ELLIS Unit Linz, LIT AI Lab and Institute for Machine Learning, JKU Linz, Austria · Institute of Artificial Intelligence, Center for Medical Data Science, Medical University Vienna · Clinical Research Center for Medical AI, JKU Linz, Austria
BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases
Jun 23, 2026Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi +14Distribution Shift Medical Image Analysis
Johns Hopkins University · German Cancer Research Center · University Hospital Basel +10
Adversarial Batch Representation Augmentation for Batch Correction in High-Content Cellular Screening
Mar 5, 2026Lei Tong, Xujing Yao, Adam Corrigan +6Batch Effect Correction Domain Generalization
Data Sciences and Quantitative Biology, Discovery Sciences, AstraZeneca R&D, Cambridge, UK · School of Computing and Mathematical Sciences, University of Leicester, Leicester, UK · College of Computer and Information Engineering, Nanjing Tech University, Nanjing, China +1

Apr 22, 2026J/K move · Enter open · S save

Stabilizing In-Context Multi-Source Domain Adaptation for Biomedical Images Through Controls

Ana Sanchez-Fernandez, Thomas Pinetz, Werner Zellinger, Günter Klambauer

ELLIS Unit Linz, LIT AI Lab and Institute for Machine Learning, JKU Linz, Austria · Institute of Artificial Intelligence, Center for Medical Data Science, Medical University Vienna · Clinical Research Center for Medical AI, JKU Linz, Austria

Multi-Source Domain Adaptation Batch Effect Correction Test-Time Adaptation Domain Adaptation

Open paper AskPDF arXiv

Biomedical imaging data presents enormous potential for deep learning models to predict invaluable properties, such as diseases and drug effects. However, unavoidable alterations of the technical conditions cause batch effects: variations between groups of samples that are not due to any biological signal of interest. Batch effects greatly hinder the generalization abilities of deep learning models, preventing their practical use in the real world. Unsupervised Domain Adaptation (UDA) methods have been proposed to mitigate batch effects, but they usually assume that the data is comprised of only one source domain and one target domain, whereas biological datasets are comprised of multiple domains, both at training and at inference time. While Batch Normalization-based test-time and meta-learning adaptation methods offer a promising mechanism for domain alignment, we show that existing approaches exhibit degraded performance under the usual inference scenarios of small target batch sizes and label shift. We address these limitations by leveraging negative control samples, which are consistently present in every experimental batch in biological datasets, as stable context for adaptation. We propose CS-ARM-BN, a meta-learning BN adaptation method that uses controls both during training and inference to stabilize domain statistics. We perform a suite of experiments of Mechanism-Of-Action (MoA) classification, a crucial task for drug discovery, on the large JUMP-CP imaging dataset. Our experiments show that CS-ARM-BN substantially improves robustness to batch size and class distribution shifts, enabling practical use of deep learning models for biomedical images.

Abstract

Distribution Shift Robustness

Explore similar work

Stabilizing In-Context Multi-Source Domain Adaptation for Biomedical Images Through Controls

BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases

Adversarial Batch Representation Augmentation for Batch Correction in High-Content Cellular Screening