A Standardized Framework For Evaluating Gene Expression Generative Models

Andrea Rubbi; Andrea Giuseppe Di Francesco; Mohammad Lotfollahi; Pietro Liò

q-bio.GNMar 11, 2026

A Standardized Framework For Evaluating Gene Expression Generative Models

Authors: Andrea Rubbi, Andrea Giuseppe Di Francesco, Mohammad Lotfollahi, Pietro Liò

Organizations: Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom · Wellcome Sanger Institute, Cambridge, United Kingdom · Sapienza University of Rome, Rome, Italy · ISTI-CNR, Institute of Information Science and Technologies, Pisa, Italy ·

Abstract

The rapid development of generative models for single-cell gene expression data has created an urgent need for standardised evaluation frameworks. Current evaluation practices suffer from inconsistent metric implementations, incomparable hyperparameter choices, and a lack of biologically-grounded metrics. We present Generated Genetic Expression Evaluator (GGE), an open-source Python framework that addresses these challenges by providing a comprehensive suite of distributional metrics with explicit computation space options and biologically-motivated evaluation through differentially expressed gene (DEG)-focused analysis and perturbation-effect correlation, enabling standardized reporting and reproducible benchmarking. Through extensive analysis of the single-cell generative modeling literature, we identify that no standardized evaluation protocol exists. Methods report incomparable metrics computed in different spaces with different hyperparameters. We demonstrate that metric values vary substantially depending on implementation choices, highlighting the critical need for standardization. GGE enables fair comparison across generative approaches and accelerates progress in perturbation response prediction, cellular identity modeling, and counterfactual inference.

Figures & tables

Figure 1: GGE is a unified framework for the evaluation of generated genetic expression profiles. GGE aims to standardise the otherwise heterogeneous set of evaluation protocols proposed by the multitude of generative models implemented for this task.

Method	MSE	$\bm{R^{2}}$	$\bm{W_{1}/W_{2}}$	MMD	Corr	DEG-specific	Space
scVI ( Lopez et al., 2018 )	✓	✓	—	—	—	—	Raw
scGen ( Lotfollahi et al., 2019 )	✓	✓	✓	✓	✓	Top-100 DEG	Raw
Waddington-OT ( Schiebinger et al., 2019 )	—	—	✓	—	—	—	PCA-30
CellOT ( Bunne et al., 2023 )	✓	—	✓	—	—	—	PCA-50
CPA ( Lotfollahi et al., 2023 )	✓	✓	—	—	✓	Top-50 DEG	Raw
GEARS ( Roohani et al., 2024 )	✓	—	—	—	✓	Top-20 DEG	Raw/HVG

Table 1: Survey of evaluation metrics used by single-cell generative models. “Space” indicates whether metrics are computed in raw gene space (Raw), PCA space (PCA-

k

), or on highly variable genes (HVG). Dashes indicate the metric was not reported. Recent methods (STATE, CellFlow) show a trend toward more comprehensive evaluation, yet protocols remain inconsistent.

Configuration	$W_{2}$	Energy	Time (s)
Raw ( $G=2000$ )	$104.3\pm 0.3$	$0.65\pm 0.01$	1.9
PCA-100	$53.8\pm 0.1$	$0.20\pm 0.00$	3.0
PCA-50	$33.6\pm 0.1$	$0.09\pm 0.00$	2.7
PCA-25	$17.2\pm 0.1$	$0.09\pm 0.00$	2.4

Table 2: Distributional metric values under different computation spaces. The same data produces

\sim

5–10

\times

different metric values depending on space and dimensionality. All metrics computed using GGE v0.6.2 defaults on MPS(M3).

Selection Strategy	Avg # DEGs	Pearson	Spearman
Top-20 (like GEARS)	20	$0.614\pm 0.066$	$0.600\pm 0.043$
Top-100 (like scGen)	100	$0.594\pm 0.024$	$0.619\pm 0.041$
Strict (lfc $>$ 1, p $<$ 0.01)	$15.3\pm 5.1$	$0.506\pm 0.217$	$0.370\pm 0.182$
Relaxed (lfc $>$ 0.25, p $<$ 0.1)	$71.7\pm 6.9$	$0.622\pm 0.079$	$0.609\pm 0.048$

Table 3: Effect of DEG selection strategy on correlation metrics. Top-N selection provides consistent gene counts across conditions; threshold-based selection adapts to perturbation strength. We benchmark on the Norman dataset (39k cells

\times

5000 genes) with 138 perturbation conditions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Feature	GGE	cell-eval
Python API	✓ (single call)	✓ (multi-step)
CLI	✓	✓
Explicit space control	✓	—
Configurable DEG thresholds	✓	—
Top-N DEG selection	✓	—
GPU acceleration (CUDA)	✓	—

Appendix

Table 4: Comparison of GGE and cell-eval design choices.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses
Sep 26, 2026Sikai Huang, Zhiwen Yang, Kai Yu +2Benchmark Design Single-Cell Perturbation Modeling
Fujian Agriculture and Forestry University · The Hong Kong Polytechnic University · Westlake University
A Pitfall in the Evaluation of Gene Perturbation Models
Oct 19, 2026George Panagopoulos, Saman Omidi, Tsz Pan Tong +1Single-Cell Perturbation Modeling Differential Testing
Department of Computer Science, University of Luxembourg, 2 Av. de l’Universite, 4365 Belval Esch-sur-Alzette, Luxembourg
SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention
Apr 18, 2026Jiahao Li, Jiayi Dong, Peng Ye +3Single-Cell Perturbation Modeling Conditional Generative Modeling
College of Computer Science and Artificial Intelligence, Fudan University · Shanghai Key Laboratory of Intelligent Information Processing, Fudan University

Sep 26, 2026J/K move · Enter open · S save

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Sikai Huang, Zhiwen Yang, Kai Yu, Jiayuan Chen+1

Fujian Agriculture and Forestry University · The Hong Kong Polytechnic University · Westlake University

Benchmark Design Single-Cell Perturbation Modeling Single-Cell Transcriptomics

Open paper AskPDF arXiv

Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.

Abstract

Synthetic Data Evaluation

Explore similar work

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

A Pitfall in the Evaluation of Gene Perturbation Models

SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention