q-bio.GNFeb 18, 2026
SaveParameter-free representations outperform single-cell foundation models on downstream benchmarks
Organizations: Department of Physics, Boston University, Boston, MA, 02215, USA · Faculty of Computing and Data Science, Boston University, Boston, MA, 02215, USA
Abstract
Single-cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use transformer-based architectures to learn a generative model for gene expression by embedding genes into a latent vector space. These embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple benchmarks commonly used to evaluate single-cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of cell identity can be captured by simple linear representations of single cell gene expression data.
Figures & tables
Figure 1: Downstream Tasks/Benchmarks analyzed in this paper . A . Cross species cell annotation. The goal of this task is to use labeled cells from one species (e.g. humans) to annotate cell types in another species. B . Discrimination between healthy and infected cells. C . Cell type classification. D . Extracting biological context (i.e. gene-TF interactions) from data.
Figure 2: Cross-species transfer learning on novel organisms and cell types. A . Pipeline used by parameter free , linear algebra-based method scTOP [ 31 ] to perform cross-species annotation on the spermatogenesis dataset [ 44 , 27 ] . B . Transfer matrix of macro F1 scores for testis cell type classification across mammals for the TranscriptFormer foundation models TF-Exemplar and TF-Metazoa as reported in [ 27 ] . C . Transfer matrix of macro F1 scores for testis cell type classification using scTOP. D . Comparison between scTOP and foundation models on the cross-species annotation task (F1 scores for foundation models are reported in [ 27 ] ).
Figure 3: Biological context from data . Cosine similarities for embeddings from TF-Metazoa and scTOP for different male germline developmental lineages and species. A . TranscriptFormer germline; B . TranscriptFormer species; C . scTOP germline; and D . scTOP species. E . Cosine similarity from scTOP between humans and indicated species as a function of evolutionary distance.
Figure 4: Tabula Sapiens 2.0 cell type classification task . A : Overview of our pipeline. B : Per-tissue results for Tabula Sapiens 2.0 classification task for pipeline. C : Detailed results and comparisons between our pipeline and foundational models as reported in [ 29 ] . D : Average result per-tissue-per-type cell classification for Tabula Sapiens 2.0. E : F1-scores distribution per cell type for our pipeline.
Figure 5: Identifying cells infected by SARS-CoV-2. A . Schematic illustrating why local classifiers are necessary for this task. B . Comparison between our pipeline (yellow) and foundation models (as reported in [ 27 ] ) at classifying SARS-CoV-2 infected and uninfected cell from four distinct donors (data from [ 48 ] ). C . Comparison of average disease state prediction F1 scores of uninfected and infected cells across all tissues and donors for our pipeline and foundation models.
Figure 6: Validation of inferred TF-gene associations . Each point represents the Pointwise Mutual Information (PMI)-enrichment score between the indicated transcription factor (row name) and a protein coding gene. PMI scores are calculated using normalized scRNA-seq gene expression profiles from scTOP. Points lying above that dashed line indicate predicted interactions that are statistically significant after FDR correction, with points colored red indicating a protein-protein interactions that has been validated using the STRING database [ 49 ] .
Figure 7: Detecting non-linear structure in data using geodesic distances from Isomap . A . Swiss roll dataset in three dimensions. B . Two dimensional projections of the swiss roll dataset using Principle Component Analysis (PCA) and the non-linear manifold learning technique Isomap [ 50 ] which uses the geodesic distance between points to create embeddings. C . Schematic illustrating difference between geodesic and Euclidean distance. D . Pearson correlation between Euclidean distances and geodesic distances between data points in the swiss roll dataset as a function of the Isomap hyper-parameter (number of nearest-neighbors).
Figure 8: Single-cell scRNA-seq datasets are approximately linear . A . Two dimensional embeddings of the human spermatogenesis dataset [ 44 , 27 ] using Principle Component Analysis (PCA) and Isomap. B . Correlation between geodesics and Euclidean distance as a function of the Isomap hyper-parameter (number of nearest-neighbors). This correlation is higher than 0.9, providing strong evidence that the subspace spanned by the human spermatogenesis dataset is approximately linear.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Macro F1-scores for each choice of number of ANOVA selected genes and number of principal components. We notice that most choices lead to similar results.
Figure 10: Worst F1-scores per cell types.
Figure 11: scTOP performance on Tabula Sapiens 2.0. A : Macro F1-score of scTOP alone compared to foundation-model baselines. B : Macro F1-score after adding ANOVA-based gene selection. C : Per-tissue macro F1-scores using scTOP alone. D : Per-tissue macro F1-scores for scTOP with ANOVA selection.
Figure 12: Detailed tissue-level performance. A : scTOP alone across five representative tissues. B : scTOP with ANOVA-based gene selection on the same tissues.
Figure 13: Comparison between the cost for extracting embeddings using TranscriptFormer models and pre-processing data using scTOP. The error bars indicate that the results are estimates and depends on the size of the model used (such as TF-exemplar, TF-sapiens, and TF-Metazoa). A : Floating point operations per-cell comparison. scTOP pre-processing is to times cheaper than extracting embeddings with TranscriptFormer. B : scTOP is to faster in processing cells than TranscriptFormer. Notice however that scTOP uses CPU while TranscriptFormer uses GPU, i.e. is also much cheaper.
Figure 14: CO 2 cost comparison between scTOP and TranscriptFormer during inference.
Figure 15: Monetary comparison between scTOP and TranscriptFormer during inference.
Figure 16: Manifold analyses similar to the main text for the other seven spermatogenesis datasets.
Figure 17: Manifold analyses across four different tissues on Tabula Sapiens 2.0.
Figure 18: A : Confusion matrix for heart tissue. B : Distance between centroids across cell types.
Figure 19: PCA study of smooth muscle cells . A : Smooth muscle cells PC1 and PC2. B : PCA for smooth muscle cells colored by correct and incorrect annotation. C : Smooth muscle cells colored by number of genes with at least one count. D : Training samples PC1 and PC2 for smooth muscle cells and cell types that it gets confused to. E : Smooth muscle cells PC1 and PC2 colored by predicted label for it.
Figure 20: LDA analyses . A : PC1 and PC2 projections of spleen tissue cells after using LDA show separability of different cell types after use of our pipeline. B : PC1 and PC2 of monocyte cells show separation across tissues after the use of LDA.
More in this topic
Single-Cell RNA Sequencing
Single-cell RNA sequencing measures gene expression in individual cells to map cell types, states, and responses across tissues and conditions.
See all 38 papers