Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking

Ihor Kendiukhov

q-bio.MNMar 3, 2026

Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking

Authors: Ihor Kendiukhov

Organizations: Department of Computer Science, University of Tübingen, Germany

Abstract

Benchmark rankings are routinely used to justify scientific claims about method quality in gene regulatory network (GRN) inference, yet the stability of these rankings under plausible evaluation protocol choices is rarely examined. We present a systematic diagnostic framework for measuring ranking instability under protocol shift, including decomposition tools that separate base rate effects from discrimination effects. Using existing single cell GRN benchmark outputs across three human tissues and six inference methods, we quantify pairwise reversal rates across four protocol axes: candidate set restriction (16.3 percent, 95 percent CI 11.0 to 23.4 percent), tissue context (19.3 percent), reference network choice (32.1 percent), and symbol mapping policy (0.0 percent). A permutation null confirms that observed reversal rates are far below random order expectations (0.163 versus null mean 0.500), indicating partially stable but non invariant ranking structure. Our decomposition reveals that reversals are driven by changes in the relative discrimination ability of methods rather than by base rate inflation, a finding that challenges a common implicit assumption in GRN benchmarking. We propose concrete reporting practices for stability aware evaluation and provide a diagnostic toolkit for identifying method pairs at risk of reversal.

Figures & tables

Protocol axis	Reversals	Rate (%)	95% CI
Candidate-set shift	22/135	16.3	11.0–23.4
Tissue shift	26/135	19.3	13.5–26.7
Reference-network shift	34/106	32.1	24.0–41.5
Mapping-policy shift	0/165	0.0	0.0–2.3

Table 1: Summary of pairwise ranking reversal rates across four protocol-shift axes.

Figure 1: Pairwise reversal rates under candidate-set shifts, stratified by tissue and shift type. Each cell shows the fraction of method pairs whose ranking reverses. Immune evaluations exhibit the highest sensitivity to candidate-set restriction.

Figure 2: Decomposition of margin shifts into base-rate and discrimination components. Reversal cases (red) show that discrimination changes, rather than base-rate inflation, drive rank flips.

Figure 3: Pairwise reversal rates under tissue shifts, stratified by candidate-set type. More constrained candidate spaces amplify cross-tissue ranking instability.

Figure 4: Pairwise reversal rates under reference-network shifts in immune baseline evaluations. Different reference networks encode different biological evidence classes, producing high ranking instability.

Figure 5: Observed candidate-shift reversal rate versus permutation null distribution (5,000 permutations). The observed rate is far below random-order expectations, indicating substantial shared ranking structure that coexists with nontrivial instability.

Figure 6: Instability screening performance across quantile thresholds under leave-one-tissue-out cross-validation. The criterion provides useful high-recall triage at the cost of moderate precision.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 391 papers

When Does Gene Regulatory Network Inference Break? A Controlled Diagnostic Study of Causal and Correlational Methods on Single-Cell Data
May 6, 2026Miguel Fernandez-de-Retana, Ruben Sanchez-Corcuera, Unai Zulaika +2Causal Discovery Gene Regulatory Network Inference
Faculty of Engineering University of Deusto Bilbao, Spain
CoDiffGRN: Rethinking Gene Regulatory Network Inference via the BEELINE-KGC Benchmark and Co-evolutionary Discrete Diffusion
Jul 14, 2026Jiaze Song, Runhao Zhao, Minghao Xu +2Link Prediction Gene Regulatory Network Inference
Peking University · National University of Defense Technology
A Pitfall in the Evaluation of Gene Perturbation Models
Oct 19, 2026George Panagopoulos, Saman Omidi, Tsz Pan Tong +1Single-Cell Perturbation Modeling Differential Testing
Department of Computer Science, University of Luxembourg, 2 Av. de l’Universite, 4365 Belval Esch-sur-Alzette, Luxembourg

May 6, 2026J/K move · Enter open · S save

When Does Gene Regulatory Network Inference Break? A Controlled Diagnostic Study of Causal and Correlational Methods on Single-Cell Data

Miguel Fernandez-de-Retana, Ruben Sanchez-Corcuera, Unai Zulaika, Aritz Bilbao-Jayo+1

Faculty of Engineering University of Deusto Bilbao, Spain

Causal Discovery Gene Regulatory Network Inference Spurious Correlation Robustness Genomics Single-Cell RNA Sequencing

Open paper AskPDF arXiv

Despite theoretical advantages, causal methods for Gene Regulatory Network (GRN) inference from single-cell RNA-seq data consistently fail to match or outperform correlation-based baselines in many realistic benchmarks, a persistent puzzle which casts doubt on the value of causality for this task. We argue that existing benchmarks are insufficiently controlled to answer this question because they evaluate on real or semi-real data where multiple pathologies co-occur, confounding failure modes, and obscuring the specific conditions under which different inference methods excel or fail. To address this gap, we introduce a controlled diagnostic framework that isolates seven biologically motivated pathologies (dropout, latent confounders, cell-type mixing, feedback loops, network density, sample size, and pseudotime drift) and measure how six representative methods spanning three inference paradigms degrade as each pathology intensifies. Across 6,120 controlled experiments, we find that causal methods genuinely dominate in clean and structurally favorable regimes, but specific pathologies (notably dropout and latent confounders) selectively neutralize their advantages. We further introduce an error-type decomposition that reveals methods with similar aggregate accuracy commit qualitatively different errors. To probe whether single-pathology effects persist when multiple stressors co-occur, we perform an interaction sweep over the three most impactful pathologies and find that their joint effects are sub-additive, while also exposing density-conditional cross-overs invisible to single-dial analysis. Our findings offer a nuanced understanding of when and why different methods succeed or fail for GRN inference, providing actionable insights for method development and practical guidance for practitioners.

Abstract

Benchmark Design

Explore similar work

When Does Gene Regulatory Network Inference Break? A Controlled Diagnostic Study of Causal and Correlational Methods on Single-Cell Data

CoDiffGRN: Rethinking Gene Regulatory Network Inference via the BEELINE-KGC Benchmark and Co-evolutionary Discrete Diffusion

A Pitfall in the Evaluation of Gene Perturbation Models