Cell types are organized by taxonomic relationships that define lineages in the so-called cell ontology. However, most existing foundation models typically ignore cell-type lineages encoded in this cell ontology. We introduce a biologically informed foundation model called scOntoFM, which embeds hierarchical cell ontology into representation learning. Using Lowest Common Ancestor (LCA) distances from the ontology graph, our framework pairs efficient offline triplet sampling with hierarchical cell-ontology learning. This approach jointly preserves local neighborhoods and shapes the global embedding geometry to reflect the ontology structure. Across diverse cell- and gene-level benchmarks, the model consistently improves performance, with especially strong gains in zero-shot settings. Our model also balances batch integration with biological structure preservation, providing complementary perspectives to existing embedding evaluation frameworks and supporting scalable biological discovery.
Figures & tables
Figure 1: Proposed model workflow. (a) scOntoFM pre-training data corpus details (left), and the distribution of major tissues and diseases in the corpus (right). (b) An illustration of cell ontology graph and its derived LCA distance matrix. (c) Ontology-aware loss terms in scOntoFM, the dynamic margin α(ca,cn) is controlled by the distance between the anchor cell-type ca and negative cell-type cn , and the depth of the anchor ca in the ontology graph. Down below is the hierarchical ontology alignment loss, in which Δemb and Δ~onto∈RB×B represent correspondingly the distance matrix of our model embeddings and its corresponding LCA distance matrix in the batch (of size B ), and where rnemb and rnonto are the soft-ranking rows in their respective distance matrices. ρS(⋅,⋅) is the Spearman correlation for the soft-ranking vectors. (d) Downstream tasks of scOntoFM, from left to right and grouped into cell- and gene-level tasks as presented in the results.
Figure 2Figure 3Table 4Table 5Figure 6
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Distribution of positives per sample under online random sampling in 10% of our pretraining dataset. Pink bars show the absolute frequency of positive samples, and purple bars show the corresponding proportions.
LCA distance
1
2
3
4
5
6
7
8
9
10
Mean Expr. dist. ± sd
0.55 ± 0.37
0.72 ± 0.33
0.82 ± 0.30
0.91 ± 0.23
0.96 ± 0.20
0.99 ± 0.17
1.00 ± 0.16
1.02 ± 0.15
1.04 ± 0.13
1.06 ± 0.12
Appendix
Table 6: The relationship between ontology LCA distance and transcriptomic distance in Tabula Sapiens v2.
Figure 8: Scatter plots of LCA distance versus cell-type pairwise transcriptomic distance for each of the nine tissues.
Figure 10
Component
Configuration
Backbone Model
BERT ( BertForMaskedLM )
BERT Architecture
Vocal size: 19941
Hidden size: 256
Layers: 6
Attention heads: 4
Intermediate size: 512
Appendix
Table 7: Model architecture and training configuration.
Metric
Value
CPU
Intel Xeon Gold 6252
Time Consumption
∼ 5 minutes
Peak Memory Usage
6.5 GB
Final File Size
132 MB
Appendix
Table 8: Computational cost of constructing the Cell-Dataset Index Map.
Model
Pretraining Data Size
# Parameters
Modeling Strategy
geneformer
30M
10,316,196
MLM, no ontology
scGPT
33M
51,330,049
Auto-regressive, no ontology
cellPLM
11.4M
82,402,543
VAE
scCello
22M
10,683,654
MLM + ontology
SCimilarity
7.9M
62,332,627
Autoencoder + ontology
scOntoFM
27.5M
8,846,848
MLM + ontology
Appendix
Table 9: Model comparison in terms of pretraining data size, model scale, and modeling strategy.
Table 10: Summary of zero-shot clustering evaluation datasets used in this study.
Dataset ID
Dataset ID
Dataset ID
99e317c0-1fab-4f73-b511-859e752fd1c3
6725ee8e-ef5b-4e68-8901-61bd14a1fe73
e2808a6e-e2ea-41b9-b38c-4a08f1677f02
98f5d518-1e60-4307-b732-ab045cd3cbc2
bab7432a-5cfe-45ea-928c-422d03c45cdd
6c87755e-a671-41a8-9c7e-4e43b850a57b
e6361237-ac4e-4c5d-ad8f-f16aca0c0a8f
242c6e7f-9016-4048-af70-d631f5eea188
d5a2d011-3765-4dad-ac36-3fff087589fe
43245158-5ae1-4e71-a9a6-67eef49c26bc
a65bcc2d-4243-44c1-a262-ab7dcddfcf86
4e374783-0152-423e-bcb6-b87cc232f64d
5e7fff82-b8eb-4220-bf65-c6feb6765ea1
8d1dd010-5cbc-43fb-83f8-e0de8e8517da
b252b015-b488-4d5c-b16e-968c13e48a2c
72822932-10f6-466f-baf3-a2c1d89364bc
881fe679-c6e0-45a3-9427-c4e81be6921f
4dd1cd23-fc4d-4fd1-9709-602540f3ca6f
Appendix
Table 11: CZI query dataset id for the 212 unseen evaluation datasets.
Dataset
Model
Accuracy
Macro-F1
HIHA
scOntoFM
0.92 ± 0.00
0.77 ± 0.01
HCE-TabNet
0.91 ± 0.00
0.71 ± 0.02
HCE-MLP
0.90 ± 0.00
0.70 ± 0.00
Pancreas
scOntoFM
0.99 ± 0.00
0.97 ± 0.01
HCE-TabNet
0.98 ± 0.00
0.90 ± 0.01
HCE-MLP
0.95 ± 0.00
0.88 ± 0.01
Appendix
Table 12: Benchmark HCE model in cell-type annotation (fine-tuning) on the HIHA and Pancreas datasets. Results are reported as mean ± standard deviation.
CL ID
cellPLM
Geneformer
Ours
scCello
scGPT
SCimilarity
count
CL ID
cellPLM
Geneformer
Ours
scCello
scGPT
SCimilarity
count
CL:0000030
0.015
0.004
0.041
0.02
0.002
0
10000
CL:0009070
0
0.061
0.003
0.304
0.001
0
5888
CL:0000047
0
0
0.024
0
0
0
10000
CL:0009071
0.949
0.748
0.886
0.433
0.836
0.787
3641
CL:0000060
0
0
0.001
0
0.002
0
4451
CL:0010012
0.05
0.01
0.001
0
0
0.082
10000
CL:0000083
0
0.028
0.004
0.342
0.183
0.856
9257
CL:0011031
0.576
0.675
0.938
0.983
0.873
0.998
10000
CL:0000121
0
0.001
0
0.17
0
0
10000
CL:0019021
0.408
0.089
0.649
0.447
0.357
0.167
4533
CL:0000159
1
0.157
1
0.131
1
0.063
5088
CL:1000022
0.498
0.59
0.882
0.946
0.498
0.052
3513
Appendix
Table 13: Predicted accuracy per novel cell type across SOTA methods. The best result for each cell type is shown in bold .
Category
Specification
Input dimension
d (embedding dimension)
Hidden layer
128 units, ReLU activation
Bottleneck layer
64 units, ReLU activation
Output layer
3 units (multi-label classification)
Dropout rate
0.2 (applied after each hidden layer)
Loss function
Binary Cross-Entropy with Logits
Appendix
Table 14: Summary of the architecture and training configuration for static gene embedding GO function classifier.
Dataset
ST (spots × genes)
SC (cells × genes)
Shared genes for imputation
Lung cancer
31,520×423
207,853×20,793
422
Breast cancer
4,784×1,000
9,234×19,098
891
SEA-AD
18,085×180
15,876×20,080
132
Appendix
Table 15: Summary of spatial transcriptomics (ST) and single-cell (SC) datasets.
Dataset
Ours
Geneformer
scGPT
scCello
CellPLM
SCimilarity
HIHA
0.7533
0.4444
0.6002
0.5668
0.7037
0.7730
Heart
0.8324
0.6862
0.8326
0.8666
0.7847
0.8299
Pancreas
0.8140
0.6113
0.7967
0.7203
0.7024
0.8023
Lung
0.8467
0.8395
0.8397
0.8299
0.8379
0.8566
Liver
0.7730
0.6853
0.7886
0.7720
0.7102
0.7921
Breast TNBC
0.8724
0.6824
0.8418
0.8423
0.8377
0.8601
Appendix
Table 16: Performance comparison of zero-shot clustering scIB overall score across tissues/datasets. The best results are shown in bold for each dataset.
Dataset
Method
NMI
ARI
ASW label
GraphConn
ASW label/batch
AvgBio
AvgBatch
Overall
HIHA
Ours
0.7597
0.6422
0.6021
0.8868
0.8756
0.6680
0.8812
0.7533
Geneformer
0.2500
0.1355
0.4560
0.5117
0.8687
0.2805
0.6902
0.4444
scGPT
0.5199
0.2619
0.4977
0.8218
0.8995
0.4265
0.8607
0.6002
scCello
0.4704
0.3245
0.4824
0.6418
0.9149
0.4257
0.7784
0.5668
cellPLM
0.7101
0.4755
0.5365
0.9208
0.8755
0.5740
0.8982
0.7037
SCimilarity
0.7675
0.7375
0.5833
0.9167
0.8602
0.6961
0.8884
0.7730
Appendix
Table 17: Zero-shot clustering benchmarking results of detailed scIB metrics for each of the nine evaluation datasets. The best performance in each metric for each dataset is highlighted in bold .
Spearman
Kendall’s τ
Pearson
Dataset
Ours
Geneformer
scGPT
scCello
CellPLM
SCimilarity
Ours
Geneformer
scGPT
scCello
CellPLM
SCimilarity
Ours
Geneformer
scGPT
scCello
CellPLM
SCimilarity
HIHA
0.7157
0.3890
0.4174
0.4724
0.3772
0.6201
0.5688
0.2849
0.3171
0.3590
0.2895
0.4921
0.7669
0.4527
0.5300
0.5946
0.5215
0.7294
Heart
0.7074
0.4220
0.6605
0.4163
0.5100
0.5090
0.5783
0.3477
0.5275
0.3277
0.3929
0.4117
0.7787
0.6073
0.7914
0.6462
0.6969
0.6909
Pancreas
0.3992
0.4040
0.3978
0.3313
0.5470
0.6646
0.2960
0.3213
0.3135
0.2390
0.4454
0.5699
0.7398
0.7139
0.7660
0.6478
0.7780
0.8412
Lung
0.6777
0.6719
0.4047
0.4655
0.4624
0.5958
0.5615
0.5621
0.3270
0.3684
0.3745
0.4890
0.8381
0.8127
0.7396
0.7519
0.7474
0.7886
Liver
0.5704
0.4905
0.4824
0.4925
0.5023
0.4842
0.4691
0.3857
0.3799
0.3923
0.4008
0.3781
0.7411
0.6457
0.6423
0.6718
0.6716
0.6501
Appendix
Table 18: Dataset-wise comparison of zero-shot clustering scGraph-LCA metric under Spearman, Kendall’s τ , and Pearson correlation. Best result per dataset and per metric is shown in bold .
Spearman
Kendall’s τ
Pearson
Dataset
Ours
Geneformer
scGPT
scCello
CellPLM
Ours
Geneformer
scGPT
scCello
CellPLM
Ours
Geneformer
scGPT
scCello
CellPLM
HIHA
0.4939
0.4584
0.5000
0.5253
0.4951
0.3668
0.3292
0.3677
0.3802
0.3618
0.7998
0.6188
0.6230
0.6855
0.7118
Heart
0.3438
0.4141
0.3190
0.4372
0.3950
0.2264
0.3223
0.2132
0.3223
0.3058
0.8926
0.8406
0.8857
0.9146
0.9022
Pancreas
0.5373
0.5693
0.3700
0.6562
0.5029
0.4116
0.4697
0.2980
0.5202
0.4040
0.8831
0.8923
0.8612
0.9007
0.8579
Lung
0.5370
0.3794
0.3297
0.2933
0.2970
0.4489
0.2756
0.2311
0.2400
0.2133
0.9285
0.8861
0.9153
0.9513
0.9037
Liver
0.4974
0.2292
0.2393
0.3665
0.3649
0.3771
0.1677
0.1594
0.2552
0.2573
0.8525
0.7604
0.8307
0.8898
0.8258
Appendix
Table 19: Performance comparison across datasets of scGraph-RWR. Best result per dataset and best mean performance are shown in bold .
scIB Metrics
scGraph-LCA
scGraph-RWR
Method
NMI
ARI
ASW
Avg Bio
Spearman
Kendall’s τ
Pearson
Spearman
Kendall’s τ
Pearson
Ours
0.7804
0.3431
0.5032
0.5422
0.5114
0.3887
0.5611
0.0937
0.0569
0.4427
Geneformer
0.7213
0.2241
0.4717
0.4724
0.2962
0.2157
0.3278
0.1369
0.0916
0.3605
scGPT
0.7740
0.3014
0.5074
0.5276
0.1339
0.0962
0.2058
0.0682
0.0447
0.3638
CellPLM
0.7915
0.3334
0.5243
0.5498
0.0980
0.0687
0.1933
0.1436
0.0962
0.3980
scCello
0.7693
0.3092
0.5199
0.5328
0.2417
0.1751
0.3384
0.0670
0.0427
0.4768
Appendix
Table 20: Overall performance of zero-shot clustering of 340 in-distribution cell types in unseen datasets across SOTA methods using scIB metrics, scGraph-LCA, and scGraph-RWR. Best result per column is shown in bold .
Figure 11: (a) Clustering results of Bone marrow dataset using our model embeddings, colors represent different cell-types. (b) Distance heatmap from root centroid cell-type (hematopoietic stem cell, highlighted as yellow dot in the umap) to all the other cell-types in the dataset. (c) Cell-type centroid embedding distances heatmap for the bone marrow dataset. (d) Cell cycle scores heatmap from original paper [ 55 ] .
Figure 12: Left: LCA distance reference heatmap of scGraph-LCA in Bone marrow datast; Right: Random walk with restart (scGraph-RWR) of ontology graph distance in Bone marrow dataset.
Figure 13: (a) Umap visualization of seen cell-type centroid embedding, cell-types are colored by its ontology ancestor at depth 2. (b) Same as (a), but cell-types are colored by its ontology ancestor at depth 1. (c) Co-embedding seen centroid embeddings and unseen cell-type centroid embedding, dots are colored by seen centroid ( blue ) and unseen centroid ( orange ). (d) Our model predicted accuracy heatmap for unseen cell-types. The red circles in all the umaps are the cell-types coming from hematopoietic or myeloid cell lineage, in which the unseen cell-types (megakaryocyte progenitor cell, immature neutrophil, CD14-positive, CD16-low monocyte) in Figure 14 are selected from.
Figure 14: (a) Umap of our model embeddings on three unseen cell types and the top 5 closest seen cell types for each unseen cell type. (b) Umap visualization of seen ( orange ) and unseen ( blue ) labels. (c) Umap visualization of the three query unseen cell types. (d) LCA distance matrix of unseen cell types and the top 5 closest seen cell types for each unseen cell type.
Figure 15: (a) Our model’s umap embedding visualization colored by cell-types. (b) Violin plot of GLYCOLYSIS hallmark enrichment score distribution for each cell type. (c) Left: COAGULATION hallmark enrichment score heatmap. Right: Violin plot of COAGULATION hallmark enrichment score distribution for each cell type. (d) Left: ALLOGRAFT_REJECTION hallmark enrichment score heatmap. Right: Violin plot of ALLOGRAFT_REJECTION hallmark enrichment score distribution for each cell type.
Model
F1 Micro
F1 Macro
AP Micro
AP Macro
Ours
0.783 ± 0.01
0.632 ± 0.02
0.832 ± 0.01
0.632 ± 0.02
Geneformer
0.767 ± 0.01
0.593 ± 0.02
0.832 ± 0.01
0.613 ± 0.02
scCello
0.781 ± 0.01
0.627 ± 0.02
0.823 ± 0.01
0.620 ± 0.02
scGPT
0.809 ± 0.01
0.683 ± 0.02
0.861 ± 0.01
0.678 ± 0.02
Appendix
Table 21: GO namespace classification by aggregated gene embeddings. bold indicates the best performance and underlining denotes the second-best results
Method
Pearson ↑
Spearman ↑
Kendall’s τ↑
Ours
0.3820±0.1450
0.3429±0.1424
0.2581±0.1025
Geneformer
0.3769±0.1580
0.3388±0.1627
0.2552±0.1175
CellPLM
0.5047±0.1705
0.4251±0.1765
0.3247±0.1302
scCello
0.3286±0.1543
0.2901±0.1429
0.2167±0.1026
scGPT
0.3056±0.1470
0.2435±0.1361
0.1823±0.0980
Appendix
Table 22: Spatial gene imputation performance (mean ± std) of Pearson, Spearman, and Kendall’s τ correlations in SEA-AD dataset. Higher is better ( ↑ ). Best results are shown in bold .
Figure 16: (a) Violin plot of the distribution for Pearson, Spearman, and Kendall’s τ correlation across all the imputed genes in MERSCOPE Lung cancer 2 dataset. (b) Violin plot of the distribution for Pearson, Spearman, and Kendall’s τ correlation across all the imputed genes in breast cancer dataset. (c) Violin plot of the distribution for Pearson, Spearman, and Kendall’s τ correlation across all the imputed genes in SEA-AD dataset. Statistical significance is indicated by stars (*), with “ns” denoting no significant difference.
Model
MLM
DynamicMargin
OntoAlign
scIB ↑
scGraph-LCA ↑
MLM only
✓
✗
✗
0.7597
0.4237
w/o OntoAlign
✓
✓
✗
0.8066
0.3607
w/o DynamicMargin
✓
✗
✓
0.7841
0.7211
scOntoFM
✓
✓
✓
0.7958
0.6112
Appendix
Table 23: Ablation study of model loss components.
scIB
scGraph-LCA
Dataset
scOntoFM
w/o OntoAlign
w/o DynamicMargin
MLM only
scOntoFM
w/o OntoAlign
w/o DynamicMargin
MLM only
HIHA
0.7533
0.7745
0.7848
0.7403
0.7157
0.4282
0.8301
0.4787
Heart
0.8324
0.8549
0.8480
0.7908
0.7074
0.4171
0.6824
0.5511
Pancreas
0.8140
0.8388
0.7767
0.7191
0.3992
0.3957
0.4408
0.3351
Lung
0.8467
0.8332
0.8119
0.8091
0.6777
0.2689
0.8622
0.4425
Liver
0.7730
0.7773
0.7837
0.7553
0.5704
0.3691
0.6911
0.4847
Appendix
Table 24: Ablation study of the model’s loss terms across the nine evaluation datasets of the overall scIB score and scGraph-LCA metric.
Dataset
Reference
Target
# Ref Labels
# Target Labels
Agreement Score
HIHA
cell_type
AIFI_L1
38
9
0.259
HIHA
cell_type
AIFI_L2
38
29
0.970
HIHA
cell_type
AIFI_L3
38
71
0.830
Kidney
cell_type
SubclassLevel1
33
17
0.933
Kidney
cell_type
SubclassLevel2
33
97
0.407
Appendix
Table 25: Agreement scores between reference and target annotations across datasets.
Noise
Accuracy (mean ± std)
Macro F1 (mean ± std)
Parent (mean ± std)
Child (mean ± std)
Sibling (mean ± std)
0.1
0.941 ± 0.001
0.806 ± 0.017
0.001 ± 0.000
0.003 ± 0.001
0.014 ± 0.000
0.2
0.940 ± 0.001
0.752 ± 0.013
0.001 ± 0.000
0.003 ± 0.001
0.015 ± 0.000
0.3
0.934 ± 0.002
0.691 ± 0.008
0.001 ± 0.000
0.005 ± 0.001
0.015 ± 0.001
0.4
0.834 ± 0.036
0.572 ± 0.024
0.001 ± 0.001
0.053 ± 0.021
0.018 ± 0.010
0.5
0.150 ± 0.018
0.207 ± 0.025
0.000 ± 0.000
0.552 ± 0.002
0.035 ± 0.003
Appendix
Table 26: Impact of noise proportion on performance and hierarchical error distribution.
Figure 17: From left to right: Sankey plot of cell type annotation flow from cell_type (CZI provided ontology labels) to author annotated labels (from coarse to fine-grained) in HIHA dataset
Figure 18: From left to right: Sankey plot of cell type annotation flow from cell_type (CZI provided ontology labels) to author annotated labels (from coarse to fine-grained) in Kidney dataset
Current single-cell foundation model benchmarks universally extract final layer embeddings, assuming these represent optimal feature spaces. We systematically evaluate layer-wise representations from scFoundation (100M parameters) and Tahoe-X1 (1.3B parameters) across trajectory inference and perturbation response prediction. Our analysis reveals that optimal layers are task-dependent (trajectory peaks at 60% depth, 31% above final layers) and context-dependent (perturbation optima shift 0-96% across T cell activation states). Notably, first-layer embeddings outperform all deeper layers in quiescent cells, challenging assumptions about hierarchical feature abstraction. These findings demonstrate that "where" to extract features matters as much as "what" the model learns, necessitating systematic layer evaluation tailored to biological task and cellular context rather than defaulting to final-layer embeddings.
Vincenzo Yuto Civale, Roberto Semeraro, Andrew David Bagdanov +1
University of Siena · University of di Firenze · University of Firenze
Single-cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use transformer-based architectures to learn a generative model for gene expression by embedding genes into a latent vector space. These embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple benchmarks commonly used to evaluate single-cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of cell identity can be captured by simple linear representations of single cell gene expression data.
Huan Souza, Pankaj Mehta
Department of Physics, Boston University, Boston, MA, 02215, USA · Faculty of Computing and Data Science, Boston University, Boston, MA, 02215, USA
When biological foundation models such as scGPT and Geneformer process single-cell gene expression, what geometric and topological structure forms in their internal representations? Is that structure biologically meaningful or a training artifact, and how confident should we be in such claims? We address these questions through autonomous large-scale hypothesis screening: an AI-driven executor-brainstormer loop that proposed, tested, and refined 141 geometric and topological hypotheses across 52 iterations, covering persistent homology, manifold distances, cross-model alignment, community structure, and directed topology, all with explicit null controls and disjoint gene-pool splits. Three principal findings emerge. First, the models learn genuine geometric structure. Gene embedding neighborhoods exhibit non-trivial topology, with persistent homology significant in 11 of 12 transformer layers at p < 0.05 in the weakest domain and 12 of 12 in the other two. A multi-level distance hierarchy shows that manifold-aware metrics outperform Euclidean distance for identifying regulatory gene pairs, and graph community partitions track known transcription factor target relationships. Second, this structure is shared across independently trained models. CCA alignment between scGPT and Geneformer yields canonical correlation of 0.80 and gene retrieval accuracy of 72 percent, yet none of 19 tested methods reliably recover gene-level correspondences. The models agree on the global shape of gene space but not on precise gene placement. Third, the structure is more localized than it first appears. Under stringent null controls applied across all null families, robust signal concentrates in immune tissue, while lung and external lung signals weaken substantially.
Ihor Kendiukhov
Department of Computer Science, University of Tübingen, Tübingen, Germany