Parameter-free representations outperform single-cell foundation models on downstream benchmarks
Authors: Huan Souza, Pankaj Mehta
Organizations: Department of Physics, Boston University, Boston, MA, 02215, USA · Faculty of Computing and Data Science, Boston University, Boston, MA, 02215, USA
Single-cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use transformer-based architectures to learn a generative model for gene expression by embedding genes into a latent vector space. These embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple benchmarks commonly used to evaluate single-cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of cell identity can be captured by simple linear representations of single cell gene expression data.
Figures & tables
Figure 1: Downstream Tasks/Benchmarks analyzed in this paper . A . Cross species cell annotation. The goal of this task is to use labeled cells from one species (e.g. humans) to annotate cell types in another species. B . Discrimination between healthy and infected cells. C . Cell type classification. D . Extracting biological context (i.e. gene-TF interactions) from data.
Figure 2: Cross-species transfer learning on novel organisms and cell types. A . Pipeline used by parameter free , linear algebra-based method scTOP [ 31 ] to perform cross-species annotation on the spermatogenesis dataset [ 44 , 27 ] . B . Transfer matrix of macro F1 scores for testis cell type classification across mammals for the TranscriptFormer foundation models TF-Exemplar and TF-Metazoa as reported in [ 27 ] . C . Transfer matrix of macro F1 scores for testis cell type classification using scTOP. D . Comparison between scTOP and foundation models on the cross-species annotation task (F1 scores for foundation models are reported in [ 27 ] ).
Figure 3: Biological context from data . Cosine similarities for embeddings from TF-Metazoa and scTOP for different male germline developmental lineages and species. A . TranscriptFormer germline; B . TranscriptFormer species; C . scTOP germline; and D . scTOP species. E . Cosine similarity from scTOP between humans and indicated species as a function of evolutionary distance.
Figure 4: Tabula Sapiens 2.0 cell type classification task . A : Overview of our pipeline. B : Per-tissue results for Tabula Sapiens 2.0 classification task for pipeline. C : Detailed results and comparisons between our pipeline and foundational models as reported in [ 29 ] . D : Average result per-tissue-per-type cell classification for Tabula Sapiens 2.0. E : F1-scores distribution per cell type for our pipeline.
Figure 5: Identifying cells infected by SARS-CoV-2. A . Schematic illustrating why local classifiers are necessary for this task. B . Comparison between our pipeline (yellow) and foundation models (as reported in [ 27 ] ) at classifying SARS-CoV-2 infected and uninfected cell from four distinct donors (data from [ 48 ] ). C . Comparison of average disease state prediction F1 scores of uninfected and infected cells across all tissues and donors for our pipeline and foundation models.
Figure 6: Validation of inferred TF-gene associations . Each point represents the Pointwise Mutual Information (PMI)-enrichment score between the indicated transcription factor (row name) and a protein coding gene. PMI scores are calculated using normalized scRNA-seq gene expression profiles from scTOP. Points lying above that dashed line indicate predicted interactions that are statistically significant after FDR correction, with points colored red indicating a protein-protein interactions that has been validated using the STRING database [ 49 ] .
Figure 7: Detecting non-linear structure in data using geodesic distances from Isomap . A . Swiss roll dataset in three dimensions. B . Two dimensional projections of the swiss roll dataset using Principle Component Analysis (PCA) and the non-linear manifold learning technique Isomap [ 50 ] which uses the geodesic distance between points to create embeddings. C . Schematic illustrating difference between geodesic and Euclidean distance. D . Pearson correlation between Euclidean distances and geodesic distances between data points in the swiss roll dataset as a function of the Isomap hyper-parameter k (number of nearest-neighbors).
Figure 8: Single-cell scRNA-seq datasets are approximately linear . A . Two dimensional embeddings of the human spermatogenesis dataset [ 44 , 27 ] using Principle Component Analysis (PCA) and Isomap. B . Correlation between geodesics and Euclidean distance as a function of the Isomap hyper-parameter k (number of nearest-neighbors). This correlation is higher than 0.9, providing strong evidence that the subspace spanned by the human spermatogenesis dataset is approximately linear.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Macro F1-scores for each choice of number of ANOVA selected genes and number of principal components. We notice that most choices lead to similar results.
Figure 10: Worst F1-scores per cell types.
Figure 11: scTOP performance on Tabula Sapiens 2.0. A : Macro F1-score of scTOP alone compared to foundation-model baselines. B : Macro F1-score after adding ANOVA-based gene selection. C : Per-tissue macro F1-scores using scTOP alone. D : Per-tissue macro F1-scores for scTOP with ANOVA selection.
Figure 12: Detailed tissue-level performance. A : scTOP alone across five representative tissues. B : scTOP with ANOVA-based gene selection on the same tissues.
Figure 13: Comparison between the cost for extracting embeddings using TranscriptFormer models and pre-processing data using scTOP. The error bars indicate that the results are estimates and depends on the size of the model used (such as TF-exemplar, TF-sapiens, and TF-Metazoa). A : Floating point operations per-cell comparison. scTOP pre-processing is 105 to 106 times cheaper than extracting embeddings with TranscriptFormer. B : scTOP is 10 to 103 faster in processing cells than TranscriptFormer. Notice however that scTOP uses CPU while TranscriptFormer uses GPU, i.e. is also much cheaper.
Figure 14: CO 2 cost comparison between scTOP and TranscriptFormer during inference.
Figure 15: Monetary comparison between scTOP and TranscriptFormer during inference.
Figure 16: Manifold analyses similar to the main text for the other seven spermatogenesis datasets.
Figure 17: Manifold analyses across four different tissues on Tabula Sapiens 2.0.
Figure 18: A : Confusion matrix for heart tissue. B : Distance between centroids across cell types.
Figure 19: PCA study of smooth muscle cells . A : Smooth muscle cells PC1 and PC2. B : PCA for smooth muscle cells colored by correct and incorrect annotation. C : Smooth muscle cells colored by number of genes with at least one count. D : Training samples PC1 and PC2 for smooth muscle cells and cell types that it gets confused to. E : Smooth muscle cells PC1 and PC2 colored by predicted label for it.
Figure 20: LDA analyses . A : PC1 and PC2 projections of spleen tissue cells after using LDA show separability of different cell types after use of our pipeline. B : PC1 and PC2 of monocyte cells show separation across tissues after the use of LDA.
Current single-cell foundation model benchmarks universally extract final layer embeddings, assuming these represent optimal feature spaces. We systematically evaluate layer-wise representations from scFoundation (100M parameters) and Tahoe-X1 (1.3B parameters) across trajectory inference and perturbation response prediction. Our analysis reveals that optimal layers are task-dependent (trajectory peaks at 60% depth, 31% above final layers) and context-dependent (perturbation optima shift 0-96% across T cell activation states). Notably, first-layer embeddings outperform all deeper layers in quiescent cells, challenging assumptions about hierarchical feature abstraction. These findings demonstrate that "where" to extract features matters as much as "what" the model learns, necessitating systematic layer evaluation tailored to biological task and cellular context rather than defaulting to final-layer embeddings.
Vincenzo Yuto Civale, Roberto Semeraro, Andrew David Bagdanov +1
University of Siena · University of di Firenze · University of Firenze
We present a systematic evaluation framework - thirty-seven analyses, 153 statistical tests, four cell types, two perturbation modalities - for assessing mechanistic interpretability in single-cell foundation models. Applying this framework to scGPT and Geneformer, we find that attention patterns encode structured biological information with layer-specific organisation - protein-protein interactions in early layers, transcriptional regulation in late layers - but this structure provides no incremental value for perturbation prediction: trivial gene-level baselines outperform both attention and correlation edges (AUROC 0.81-0.88 versus 0.70), pairwise edge scores add zero predictive contribution, and causal ablation of regulatory heads produces no degradation. These findings generalise from K562 to RPE1 cells; the attention-correlation relationship is context-dependent, but gene-level dominance is universal. Cell-State Stratified Interpretability (CSSI) addresses an attention-specific scaling failure, improving GRN recovery up to 1.85x. The framework establishes reusable quality-control standards for the field.
Ihor Kendiukhov
Department of Computer Science, University of Tübingen, Tübingen, Germany
Single-cell transcriptomics enables profiling of cellular states at unprecedented resolution, but its high dimensionality, sparsity, and technical batch effects pose significant challenges for representation learning. Existing single-cell foundation models typically encode each cell independently or only model cells from the same batch for denoising, thereby underutilizing the rich relational information across batches and cell types to model gene expression patterns. We argue that single-cell models can benefit from more informative cell-context modeling. By comparing consistency and variation across cells, models can capture fine-grained gene-gene dependencies associated with cell states, which are essential for learning high-quality representations. Inspired by the use of multiple sequence alignment (MSA) context in protein modeling, we propose CellMSA, a single-cell representation learning framework that introduces an MSA-inspired inductive bias into transcriptomic modeling. For each target cell, CellMSA retrieves relevant cells from different batches and biologically related cell types as context, and summarizes cross-cell patterns into a context-dependent gene-pair representation. This representation is then injected into a pair-aware target-cell encoder for fine-grained representation learning. We pretrain CellMSA on a large-scale human single-cell corpus of approximately 109 million cell observations, including 65.6 million primary observations. Experiments show that our framework consistently outperforms existing methods across multiple benchmarks. Code is available at the following repository: https://github.com/PharMolix/CellMSA.
Suyuan Zhao, Minghao Liu, Yizhen Luo +1
Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · Tsinghua Institute of Multidisciplinary Biomedical Research (TIMBR), Tsinghua University +2