A Pitfall in the Evaluation of Gene Perturbation Models

George Panagopoulos; Saman Omidi; Tsz Pan Tong; Jun Pang

stat.APOct 19, 2026

A Pitfall in the Evaluation of Gene Perturbation Models

Authors: George Panagopoulos, Saman Omidi, Tsz Pan Tong, Jun Pang

Organizations: Department of Computer Science, University of Luxembourg, 2 Av. de l’Universite, 4365 Belval Esch-sur-Alzette, Luxembourg

Abstract

Gene perturbation models aim to predict transcriptional responses to unseen genetic interventions, but comparisons rely on heterogeneous evaluation metrics and differentially-expressed (DE) gene subsets. We investigate the consequences of these choices using eight methods across six single-gene perturbation datasets under a unified evaluation protocol. Varying the number of top DE genes reveals substantial instability in Precision-based rankings, with the winning method changing $86.7\%$ of the time, while rankings based on the rest of the metrics remain comparatively stable. Investigating this discrepancy exposes that repeating a single predicted expression profile across cell-level rows, a standard practice for many methods, can inflate significance estimates and produce excessively large predicted significant-DE sets. Specifically we found that $67.1\%$ of tested perturbation--gene pairs pass the significance threshold despite some of them having minuscule effects. We derive an inflation factor showing that replicated predictions can amplify the standardized Mann--Whitney statistic even after correction for tied ranks, without adding independent information. This artifact changes Precision mainly as the number of DE genes increases because for smaller cutoffs, most of the genes are already significant due to the effect-based ordering. Taking this pitfall into account in our benchmark comparison, we find that a random forest using Gene Ontology features achieves the best overall results.

Figures & tables

Method	Representation	Predictor	Prior	Pretrained
Bilinear	Low-dimensional factors	Bilinear regression	None	No
RF-GO	GO-derived gene features	Random forest	GO	No
Ridge-scf	scFoundation embeddings	Ridge regression	None explicit	Yes
RF-scf	scFoundation embeddings	Random forest	None explicit	Yes
GEARS	GO node embeddings	GNN	GO	No
LPM	Perturbation embeddings	NN	None explicit	No

Table 1: Overview of the evaluated methods. “Prior” denotes an explicit structured biological prior, while “Pretrained” indicates that the model or representation was learned from external data before training the benchmark-specific predictor.

Dataset	Perturbations	Mean	Median	Perturbed	Control	Genes
		Cells per perturbation		Cell counts
Jurkat ( Nadig et al., 2025 )	2,393	104.9	83.0	250,943	12,013	8,882
RPE1 ( Replogle et al., 2022 )	2,393	98.8	72.0	236,429	11,485	8,749
HepG2 ( Nadig et al., 2025 )	2,393	58.7	45.0	140,497	4,976	9,624
K562 ( Replogle et al., 2022 )	2,057	145.7	121.0	299,694	10,691	8,563
VCC ( Youngblut et al., 2025 )	150	1,220.6	1,045.0	183,097	38,176	18,080

Table 2: Dataset characteristics. Mean and median refer to cells per perturbation; perturbed and control counts include all perturbed and non-targeting control cells, respectively. “Genes” denotes the number of retained expression features.

Panel of Figure 1 — Figure 1: (a) Model rankings as a function of the number $N$ of evaluated DE genes. Each row corresponds to a dataset and each column to an evaluation metric. Rank $1$ denotes the best-performing method. Different datasets have different number of DE genes. (b) Mean absolute rank displacement relative to the ordering obtained at $N=50$ , averaged across dataset–fold blocks. (c) Frequency with which the top-ranked method changes relative to the ordering obtained at $N=50$ . The final cutoff corresponds to $4000$ genes or the maximum available number of eligible DE genes when fewer are present.

	MSE	Pearson	Precision	Overlap
Bilinear	14.31	47.70	6.15	6.09
LPM	25.14	26.70	4.21	4.42
Ridge-scf	12.32	45.96	6.06	6.37
GEARS	16.53	45.21	6.29	6.33
AVG Ctrl	14.59	33.23	4.15	4.69
AVG Train	12.40	48.92	6.15	6.65

Table 3: Model performance across evaluation metrics. All values

\times 100

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Overview of dataset characteristics. (a) Number of perturbations per dataset. (b) Mean vs. median cells per perturbation, illustrating the right-skewed coverage typical of single-cell perturbation screens. (c) Total cells per dataset, split into perturbed and non-targeting control cells. (d) Number of genes profiled per dataset.

Panel of Figure 3 — Figure 3: (a) Agreement among metric-induced model rankings as a function of the number $N$ of evaluated DE genes (mean Kendall’s $\tau_{b}$ , averaged across folds within each dataset and then across datasets with equal weight). Detection $=\{$ Overlap, Precision $\}$ , reconstruction $=\{$ MSE and Pearson $\}$ . (b) Pairwise Kendall’s $\tau_{b}$ between metric-induced model rankings at representative values of $N$ . Black rectangles indicate metric families. Values are averaged across folds within each dataset and then across datasets with equal weight.

Figure 4: Heatmap showing the rank of each evaluated model. Lower ranks indicate better performance. Black cells indicate that no DE genes were available for the corresponding value of $N$ .

Single-Cell Perturbation Modeling

Single-cell perturbation modeling predicts how genetic or chemical interventions change cellular states across cells and contexts.

See all 61 papers

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses
Sep 26, 2026Sikai Huang, Zhiwen Yang, Kai Yu +2Benchmark Design Single-Cell Perturbation Modeling
Fujian Agriculture and Forestry University · The Hong Kong Polytechnic University · Westlake University
Stable-Shift: Biologically Structured Prediction of Transcriptional Responses to Unseen Gene Perturbations
Jun 22, 2026Sajib Acharjee Dip, Liqing ZhangSingle-Cell Perturbation Modeling Genomics
Department of Computer Science, Virginia Tech Blacksburg, VA, USA · Fralin Biomedical Research Institute, Virginia Tech Roanoke, VA, USA · FBRI Cancer Research Center Washington, DC, USA
Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation
Mar 7, 2026Andrea Giuseppe Di Francesco, Andrea Rubbi, Rishabh Jain +1Single-Cell Perturbation Modeling Retrieval-Augmented Generation
University of Cambridge Wellcome Sanger Institute

Sep 26, 2026J/K move · Enter open · S save

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Sikai Huang, Zhiwen Yang, Kai Yu, Jiayuan Chen+1

Fujian Agriculture and Forestry University · The Hong Kong Polytechnic University · Westlake University

Benchmark Design Single-Cell Perturbation Modeling Single-Cell Transcriptomics

Open paper AskPDF arXiv

Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.

Abstract

Single-Cell Perturbation Modeling

Explore similar work

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Stable-Shift: Biologically Structured Prediction of Transcriptional Responses to Unseen Gene Perturbations

Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation