AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Sikai Huang; Zhiwen Yang; Kai Yu; Jiayuan Chen; Stan Z. Li

cs.AISep 26, 2026

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Authors: Sikai Huang, Zhiwen Yang, Kai Yu, Jiayuan Chen, Stan Z. Li

Organizations: Fujian Agriculture and Forestry University · The Hong Kong Polytechnic University · Westlake University

Abstract

Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.

Figures & tables

Figure 1: AmbiModBench overview. (a) Benchmark design: three deficiencies, the corresponding operations and the evidence they return. The paired contrast scores a prediction and the training-mean reference $c$ against the same held-out response $y_{t}$ ; the gene-permutation control permutes gene coordinates; the coverage operation pairs the test-to-training radius $\delta_{t}$ with the ambiguity modulus. (b) Top-100 direction-agreement gains over the training mean for kernel ridge and ridge across six RPE1 splits. (c) Gene-identity screen: permutation-induced score separation relative to the reference-to-mean gap, for seven K562 readouts; energy distance and Wasserstein distance $W_{1}$ remain at zero. (d) The same paired gain as the retained training fraction grows from 10% to 100%. (e) Kernel-ridge effects from (b) with 95% target-cluster bootstrap intervals. All evaluations use target-disjoint holdouts.

Dataset	Targets	Genes	Splits	Response scale
K562	490	1,000	15	log1p mean-count difference
RPE1	1,089	4,063	6	mean-count difference
Norman	105	5,045	3	log-normalized difference

Table 1: Benchmark datasets and evaluation regimes.

Figure 2: RPE1 directional gains reproduce across held-out splits. Effects are in percentage points (pp) relative to the training mean; positive values indicate a gain. (a) Points show paired top-100 effects for each split, with 95% target-cluster bootstrap intervals. Diamonds show pooled effects for all six splits and for the five additional splits excluding seed42. Each split has 218 test targets. Pooled intervals cluster by target across repeated appearances. (b) Empirical cumulative distributions of target-specific effects, averaged over available test appearances for each of 792 targets. These descriptive distributions weight each unique target equally, while panel (a) pools target-split records.

Figure 3: Calibration identifies which response information a score uses. (a) K562 squared error along a mixture of the prediction-side cohort mean and an independent-replicate reference, with its minimum at weight 0.2. (b) Pearson correlation and top-20 recall increase along the same mixture. Each curve is normalized by its own endpoint gain. (c) Score separation after permuting gene coordinates, divided by the score gap between the replicate reference and the cohort mean. Zero denotes invariance to gene assignment. Calibration uses all 490 K562 TFs, prediction replicates 0–2, evaluation replicates 3–4 and five gene permutations per target, reused for both evaluation replicates. Curves show descriptive cohort averages. The cohort mean is used for measurement calibration, separately from the held-out-target evaluation.

Figure 4: Increasing the observed training set strengthens the RPE1 directional signal. Gains are in percentage points (pp) relative to the refitted training mean. Test targets are fixed within each of five splits. Pale points show all five subset-ordering repeats per split at each fraction. Gray trajectories average the repeats within each split. Colored points and ribbons show pooled paired effects and 95% target-cluster bootstrap intervals after averaging repeats within target and split. Kernel ridge (a) and ridge (b) use the same nested training subsets, with all transformations and references refitted.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Regularization $\alpha$	Kernel coefficient $\gamma$
RPE1	$\{0.001,0.1,1\}$	$\{0.005,0.02,0.1,0.5\}$
K562	$\{0.001,0.1,1,10\}$	$\{0.05,0.2,1,5\}$
Norman	$\{0.001,0.1,1\}$	$\{0.05,0.2,1\}$

Appendix

Table 2: Kernel-ridge search grids. Hyperparameters are selected from training targets, without access to held-out direction scores.

Split scope	Selection	Kernel ridge	Ridge
All six	Top 100	$2.31\;[1.45,3.18]$	$1.60\;[0.76,2.44]$
All six	Top 2%	$2.45\;[1.53,3.38]$	$1.73\;[0.82,2.63]$
All six	Top 20	$3.57\;[2.44,4.69]$	$2.86\;[1.72,3.99]$
Five additional	Top 100	$2.36\;[1.49,3.26]$	$1.56\;[0.71,2.42]$
Five additional	Top 2%	$2.50\;[1.55,3.46]$	$1.68\;[0.76,2.60]$
Five additional	Top 20	$3.64\;[2.48,4.81]$	$2.83\;[1.68,3.98]$

Appendix

Table 3: RPE1 primary paired effects in percentage points with 95% target-cluster bootstrap intervals. The all-six analysis contains 1,308 records from 792 targets. The five-additional analysis contains 1,090 records from 724 targets. Positive entries indicate a gain over the training mean, and the same convention applies to the subsequent result tables.

Predictor	Selection	Score (%)	Paired effect (pp)
Training mean	Top 100	79.43	Reference
Training mean	Top 2%	79.10	Reference
Training mean	Top 20	78.94	Reference
Zero	Top 100	50.00	$-29.43\;[-30.87,-27.96]$
Zero	Top 2%	50.00	$-29.10\;[-30.62,-27.57]$
Zero	Top 20	50.00	$-28.94\;[-30.63,-27.27]$

Appendix

Table 4: RPE1 complete direction-score results. Absolute scores are percentages. Paired effects are percentage points with 95% target-cluster intervals. The training mean is the reference for every paired comparison.

Predictor	Selection	Score (%)	Paired effect (pp)
Training mean	Top 100	64.42	Reference
Training mean	Top 2%	65.82	Reference
Training mean	Top 20	65.82	Reference
Zero	Top 100	50.00	$-14.42\;[-16.21,-12.61]$
Zero	Top 2%	50.00	$-15.82\;[-17.76,-13.85]$
Zero	Top 20	50.00	$-15.82\;[-17.76,-13.85]$

Appendix

Table 5: K562 complete direction-score results. Absolute scores are percentages. Paired effects are percentage points with 95% target-cluster intervals. The training mean is the reference for every paired comparison.

Predictor	Selection	Score (%)	Paired effect (pp)
Training mean	Top 100	76.95	Reference
Training mean	Top 2%	77.12	Reference
Training mean	Top 20	75.79	Reference
Zero	Top 100	50.00	$-26.95\;[-32.34,-21.12]$
Zero	Top 2%	50.00	$-27.12\;[-32.48,-21.32]$
Zero	Top 20	50.00	$-25.79\;[-31.90,-19.60]$

Appendix

Table 6: Norman complete direction-score results. Absolute scores are percentages. Paired effects are percentage points with 95% target-cluster intervals. The training mean is the reference for every paired comparison.

Training (%)	Targets	Kernel ridge	Ridge
10	87	$0.01\;[-0.52,0.54]$	$0.19\;[-0.26,0.64]$
25	218	$1.33\;[0.73,1.93]$	$1.04\;[0.48,1.59]$
50	436	$1.72\;[0.97,2.45]$	$1.26\;[0.53,1.97]$
75	653	$2.08\;[1.25,2.90]$	$1.40\;[0.60,2.18]$
100	871	$2.38\;[1.50,3.25]$	$1.56\;[0.70,2.39]$

Appendix

Table 7: Fixed-test RPE1 top-100 effects in percentage points with 95% target-cluster bootstrap intervals. Five subset repeats are averaged within target and split before inference. Each fraction includes 724 unique test targets across five splits.

Weight	MSE	Pearson	Top-20 recall
0	0.009975	-0.0257	0.0770
0.02	0.009942	-0.0146	0.0776
0.05	0.009898	0.0011	0.0891
0.1	0.009839	0.0232	0.1048
0.2	0.009777	0.0522	0.1216
0.35	0.009825	0.0737	0.1303

Appendix

Table 8: Independent-replicate K562 calibration on all 490 targets. Entries are descriptive means over targets and evaluation partitions. The mixture weight controls the noisy reference contribution.

Configuration	Paired effect [CI]	Resid. corr. [CI]	Amp. ratio
Frozen configuration	$+0.00002\;[-0.00006,+0.00010]$	$-0.021\;[-0.030,-0.004]$	$0.95\%$
Auxiliary penalty removed	$+0.00010\;[+0.00004,+0.00016]$	$+0.202\;[+0.165,+0.213]$	$0.51\%$
Early stopping removed	$+0.00009\;[-0.00004,+0.00022]$	$-0.026\;[-0.046,-0.015]$	$1.76\%$

Appendix

Table 9: Three-arm ablation of the response network’s training configuration on the six RPE1 splits (three seeds per arm). Brackets show 95% target-cluster bootstrap intervals. Paired effects are differences of the top-100 direction agreement against the split-level train-mean constant, on the fraction scale (the other tables report percentage points). The residual–truth correlation is the median across targets of the per-target correlation between the residual (prediction minus constant) and the truth residual. The amplitude ratio is the per-target ratio of residual standard deviation (network minus constant) to residual standard deviation (truth minus constant), summarized as the median across targets. The ablation covers RPE1 only, and its frozen configuration re-runs the setting of Section 5.1 with three fit seeds rather than reporting the locked single-seed estimates.

Dataset	Unique targets	Kernel ridge	Ridge
RPE1	792	$2.45\;[1.53,3.38]$	$1.73\;[0.82,2.63]$
K562	432	$-0.16\;[-0.58,0.27]$	$0.06\;[-0.19,0.32]$
Norman	49	$1.38\;[-1.38,4.36]$	$0.88\;[-0.74,2.66]$

Appendix

Table 10: Coordinated top-2% direction-agreement gains over the training mean, in percentage points. Brackets show 95% target-cluster bootstrap intervals. The score selects at least 20 genes. Target populations and preprocessing remain dataset specific (Table 1 ).

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 391 papers

Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation
Mar 7, 2026Andrea Giuseppe Di Francesco, Andrea Rubbi, Rishabh Jain +1Single-Cell Perturbation Modeling Retrieval-Augmented Generation
University of Cambridge Wellcome Sanger Institute
DeMixPert: Decomposed Response Modeling with Gaussian Mixtures for OOD Single-Cell Perturbation Prediction
Aug 24, 2026Jiawen Liu, Xuechenxiao Cao, Yutong Li +5Single-Cell Perturbation Modeling Single-Cell Transcriptomics
School of Computer Science and Engineering, South China University of Technology, Guangzhou, China · School of Future Technology, South China University of Technology, Guangzhou, China
A Pitfall in the Evaluation of Gene Perturbation Models
Oct 19, 2026George Panagopoulos, Saman Omidi, Tsz Pan Tong +1Single-Cell Perturbation Modeling Differential Testing
Department of Computer Science, University of Luxembourg, 2 Av. de l’Universite, 4365 Belval Esch-sur-Alzette, Luxembourg

Mar 7, 2026J/K move · Enter open · S save

Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation

Andrea Giuseppe Di Francesco, Andrea Rubbi, Rishabh Jain, Pietro Liò

University of Cambridge Wellcome Sanger Institute

Single-Cell Perturbation Modeling Retrieval-Augmented Generation Adaptive Retrieval

Open paper AskPDF arXiv

Predicting transcriptional responses to genetic perturbations is fundamental to functional genomics and therapeutic discovery. Recent deep learning models have shown promise in single-cell perturbation response prediction, but they typically generate each response in isolation, without explicitly leveraging related perturbations. We introduce PT-RAG (Perturbation-aware Two-stage Retrieval-Augmented Generation), a plug-in retrieval-and-conditioning module for generative cellular perturbation response. PT-RAG augments an existing perturbation-response backbone with learned access to related perturbation contexts. The key challenge is that relevance is not fixed in this setting: functionally related genes may elicit different effects across cell types. PT-RAG addresses this with a two-stage retrieval mechanism: GenePT-based semantic retrieval first identifies K candidate perturbations, after which a differentiable Gumbel-Softmax selector adaptively selects retrieved contexts conditioned on the control cell state, the query perturbation, and each candidate perturbation. We evaluate PT-RAG on two backbones, a STATE-style generator used as a frozen random reservoir and a fully trained scGPT, across cross-cell-type and cross-perturbation generalization tasks. PT-RAG consistently improves distributional similarity and often overall predictive quality; for example, on scGPT cross-cell-type results, the 2-Wasserstein distance drops by 5.9% relative to scGPT alone. The code to reproduce our experiments is available at https://github.com/difra100/PT-RAG_NIPS.

Abstract

Benchmark Design

Explore similar work

Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation

DeMixPert: Decomposed Response Modeling with Gaussian Mixtures for OOD Single-Cell Perturbation Prediction

A Pitfall in the Evaluation of Gene Perturbation Models