Learning Biological Hierarchies in Single-Cell Foundation Models
Organizations: Duke University
Abstract
Cell types are organized by taxonomic relationships that define lineages in the so-called cell ontology. However, most existing foundation models typically ignore cell-type lineages encoded in this cell ontology. We introduce a biologically informed foundation model called scOntoFM, which embeds hierarchical cell ontology into representation learning. Using Lowest Common Ancestor (LCA) distances from the ontology graph, our framework pairs efficient offline triplet sampling with hierarchical cell-ontology learning. This approach jointly preserves local neighborhoods and shapes the global embedding geometry to reflect the ontology structure. Across diverse cell- and gene-level benchmarks, the model consistently improves performance, with especially strong gains in zero-shot settings. Our model also balances batch integration with biological structure preservation, providing complementary perspectives to existing embedding evaluation frameworks and supporting scalable biological discovery.
Figures & tables
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| LCA distance | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| Mean Expr. dist. sd | 0.55 0.37 | 0.72 0.33 | 0.82 0.30 | 0.91 0.23 | 0.96 0.20 | 0.99 0.17 | 1.00 0.16 | 1.02 0.15 | 1.04 0.13 | 1.06 0.12 |
| Component | Configuration |
| Backbone Model | BERT ( BertForMaskedLM ) |
| BERT Architecture | Vocal size: 19941 |
| Hidden size: 256 | |
| Layers: 6 | |
| Attention heads: 4 | |
| Intermediate size: 512 |
| Metric | Value |
| CPU | Intel Xeon Gold 6252 |
| Time Consumption | 5 minutes |
| Peak Memory Usage | 6.5 GB |
| Final File Size | 132 MB |
| Model | Pretraining Data Size | # Parameters | Modeling Strategy |
| geneformer | 30M | 10,316,196 | MLM, no ontology |
| scGPT | 33M | 51,330,049 | Auto-regressive, no ontology |
| cellPLM | 11.4M | 82,402,543 | VAE |
| scCello | 22M | 10,683,654 | MLM + ontology |
| SCimilarity | 7.9M | 62,332,627 | Autoencoder + ontology |
| scOntoFM | 27.5M | 8,846,848 | MLM + ontology |
| Dataset | # Cells | Source |
| Blood (HIHA) | 263,917 | https://cellxgene.cziscience.com/e/e522d2cd-7927-4e59-a4ed-064009569279.cxg/ |
| Heart | 211,061 | https://cellxgene.cziscience.com/e/d567b692-c374-4628-a508-8008f6778f22.cxg/ |
| Liver | 72,710 | https://cellxgene.cziscience.com/e/84cfa5aa-a0d1-415e-ac47-eb66c9cf8170.cxg/ |
| Kidney (Atlas V2) | 348,984 | https://cellxgene.cziscience.com/collections/9c9d04c4-8899-417f-bb6f-6107dcadf14f |
| Pancreas | 245,878 | https://cellxgene.cziscience.com/e/e51bae9a-c747-4b64-904a-4da7cda218ab.cxg/ |
| Breast TNBC | 52,681 | https://cellxgene.cziscience.com/e/9a64bf99-ebe5-4276-93a8-bee9dff1cd47.cxg/ |
| Dataset ID | Dataset ID | Dataset ID |
| 99e317c0-1fab-4f73-b511-859e752fd1c3 | 6725ee8e-ef5b-4e68-8901-61bd14a1fe73 | e2808a6e-e2ea-41b9-b38c-4a08f1677f02 |
| 98f5d518-1e60-4307-b732-ab045cd3cbc2 | bab7432a-5cfe-45ea-928c-422d03c45cdd | 6c87755e-a671-41a8-9c7e-4e43b850a57b |
| e6361237-ac4e-4c5d-ad8f-f16aca0c0a8f | 242c6e7f-9016-4048-af70-d631f5eea188 | d5a2d011-3765-4dad-ac36-3fff087589fe |
| 43245158-5ae1-4e71-a9a6-67eef49c26bc | a65bcc2d-4243-44c1-a262-ab7dcddfcf86 | 4e374783-0152-423e-bcb6-b87cc232f64d |
| 5e7fff82-b8eb-4220-bf65-c6feb6765ea1 | 8d1dd010-5cbc-43fb-83f8-e0de8e8517da | b252b015-b488-4d5c-b16e-968c13e48a2c |
| 72822932-10f6-466f-baf3-a2c1d89364bc | 881fe679-c6e0-45a3-9427-c4e81be6921f | 4dd1cd23-fc4d-4fd1-9709-602540f3ca6f |
| Dataset | Model | Accuracy | Macro-F1 |
| HIHA | scOntoFM | 0.92 0.00 | 0.77 0.01 |
| HCE-TabNet | 0.91 0.00 | 0.71 0.02 | |
| HCE-MLP | 0.90 0.00 | 0.70 0.00 | |
| Pancreas | scOntoFM | 0.99 0.00 | 0.97 0.01 |
| HCE-TabNet | 0.98 0.00 | 0.90 0.01 | |
| HCE-MLP | 0.95 0.00 | 0.88 0.01 |
| CL ID | cellPLM | Geneformer | Ours | scCello | scGPT | SCimilarity | count | CL ID | cellPLM | Geneformer | Ours | scCello | scGPT | SCimilarity | count |
| CL:0000030 | 0.015 | 0.004 | 0.041 | 0.02 | 0.002 | 0 | 10000 | CL:0009070 | 0 | 0.061 | 0.003 | 0.304 | 0.001 | 0 | 5888 |
| CL:0000047 | 0 | 0 | 0.024 | 0 | 0 | 0 | 10000 | CL:0009071 | 0.949 | 0.748 | 0.886 | 0.433 | 0.836 | 0.787 | 3641 |
| CL:0000060 | 0 | 0 | 0.001 | 0 | 0.002 | 0 | 4451 | CL:0010012 | 0.05 | 0.01 | 0.001 | 0 | 0 | 0.082 | 10000 |
| CL:0000083 | 0 | 0.028 | 0.004 | 0.342 | 0.183 | 0.856 | 9257 | CL:0011031 | 0.576 | 0.675 | 0.938 | 0.983 | 0.873 | 0.998 | 10000 |
| CL:0000121 | 0 | 0.001 | 0 | 0.17 | 0 | 0 | 10000 | CL:0019021 | 0.408 | 0.089 | 0.649 | 0.447 | 0.357 | 0.167 | 4533 |
| CL:0000159 | 1 | 0.157 | 1 | 0.131 | 1 | 0.063 | 5088 | CL:1000022 | 0.498 | 0.59 | 0.882 | 0.946 | 0.498 | 0.052 | 3513 |
| Category | Specification |
| Input dimension | (embedding dimension) |
| Hidden layer | 128 units, ReLU activation |
| Bottleneck layer | 64 units, ReLU activation |
| Output layer | 3 units (multi-label classification) |
| Dropout rate | 0.2 (applied after each hidden layer) |
| Loss function | Binary Cross-Entropy with Logits |
| Dataset | ST (spots genes) | SC (cells genes) | Shared genes for imputation |
| Lung cancer | 422 | ||
| Breast cancer | 891 | ||
| SEA-AD | 132 |
| Dataset | Ours | Geneformer | scGPT | scCello | CellPLM | SCimilarity |
| HIHA | 0.7533 | 0.4444 | 0.6002 | 0.5668 | 0.7037 | 0.7730 |
| Heart | 0.8324 | 0.6862 | 0.8326 | 0.8666 | 0.7847 | 0.8299 |
| Pancreas | 0.8140 | 0.6113 | 0.7967 | 0.7203 | 0.7024 | 0.8023 |
| Lung | 0.8467 | 0.8395 | 0.8397 | 0.8299 | 0.8379 | 0.8566 |
| Liver | 0.7730 | 0.6853 | 0.7886 | 0.7720 | 0.7102 | 0.7921 |
| Breast TNBC | 0.8724 | 0.6824 | 0.8418 | 0.8423 | 0.8377 | 0.8601 |
| Dataset | Method | NMI | ARI | ASW label | GraphConn | ASW label/batch | AvgBio | AvgBatch | Overall |
| HIHA | Ours | 0.7597 | 0.6422 | 0.6021 | 0.8868 | 0.8756 | 0.6680 | 0.8812 | 0.7533 |
| Geneformer | 0.2500 | 0.1355 | 0.4560 | 0.5117 | 0.8687 | 0.2805 | 0.6902 | 0.4444 | |
| scGPT | 0.5199 | 0.2619 | 0.4977 | 0.8218 | 0.8995 | 0.4265 | 0.8607 | 0.6002 | |
| scCello | 0.4704 | 0.3245 | 0.4824 | 0.6418 | 0.9149 | 0.4257 | 0.7784 | 0.5668 | |
| cellPLM | 0.7101 | 0.4755 | 0.5365 | 0.9208 | 0.8755 | 0.5740 | 0.8982 | 0.7037 | |
| SCimilarity | 0.7675 | 0.7375 | 0.5833 | 0.9167 | 0.8602 | 0.6961 | 0.8884 | 0.7730 |
| Spearman | Kendall’s | Pearson | ||||||||||||||||
| Dataset | Ours | Geneformer | scGPT | scCello | CellPLM | SCimilarity | Ours | Geneformer | scGPT | scCello | CellPLM | SCimilarity | Ours | Geneformer | scGPT | scCello | CellPLM | SCimilarity |
| HIHA | 0.7157 | 0.3890 | 0.4174 | 0.4724 | 0.3772 | 0.6201 | 0.5688 | 0.2849 | 0.3171 | 0.3590 | 0.2895 | 0.4921 | 0.7669 | 0.4527 | 0.5300 | 0.5946 | 0.5215 | 0.7294 |
| Heart | 0.7074 | 0.4220 | 0.6605 | 0.4163 | 0.5100 | 0.5090 | 0.5783 | 0.3477 | 0.5275 | 0.3277 | 0.3929 | 0.4117 | 0.7787 | 0.6073 | 0.7914 | 0.6462 | 0.6969 | 0.6909 |
| Pancreas | 0.3992 | 0.4040 | 0.3978 | 0.3313 | 0.5470 | 0.6646 | 0.2960 | 0.3213 | 0.3135 | 0.2390 | 0.4454 | 0.5699 | 0.7398 | 0.7139 | 0.7660 | 0.6478 | 0.7780 | 0.8412 |
| Lung | 0.6777 | 0.6719 | 0.4047 | 0.4655 | 0.4624 | 0.5958 | 0.5615 | 0.5621 | 0.3270 | 0.3684 | 0.3745 | 0.4890 | 0.8381 | 0.8127 | 0.7396 | 0.7519 | 0.7474 | 0.7886 |
| Liver | 0.5704 | 0.4905 | 0.4824 | 0.4925 | 0.5023 | 0.4842 | 0.4691 | 0.3857 | 0.3799 | 0.3923 | 0.4008 | 0.3781 | 0.7411 | 0.6457 | 0.6423 | 0.6718 | 0.6716 | 0.6501 |
| Spearman | Kendall’s | Pearson | |||||||||||||
| Dataset | Ours | Geneformer | scGPT | scCello | CellPLM | Ours | Geneformer | scGPT | scCello | CellPLM | Ours | Geneformer | scGPT | scCello | CellPLM |
| HIHA | 0.4939 | 0.4584 | 0.5000 | 0.5253 | 0.4951 | 0.3668 | 0.3292 | 0.3677 | 0.3802 | 0.3618 | 0.7998 | 0.6188 | 0.6230 | 0.6855 | 0.7118 |
| Heart | 0.3438 | 0.4141 | 0.3190 | 0.4372 | 0.3950 | 0.2264 | 0.3223 | 0.2132 | 0.3223 | 0.3058 | 0.8926 | 0.8406 | 0.8857 | 0.9146 | 0.9022 |
| Pancreas | 0.5373 | 0.5693 | 0.3700 | 0.6562 | 0.5029 | 0.4116 | 0.4697 | 0.2980 | 0.5202 | 0.4040 | 0.8831 | 0.8923 | 0.8612 | 0.9007 | 0.8579 |
| Lung | 0.5370 | 0.3794 | 0.3297 | 0.2933 | 0.2970 | 0.4489 | 0.2756 | 0.2311 | 0.2400 | 0.2133 | 0.9285 | 0.8861 | 0.9153 | 0.9513 | 0.9037 |
| Liver | 0.4974 | 0.2292 | 0.2393 | 0.3665 | 0.3649 | 0.3771 | 0.1677 | 0.1594 | 0.2552 | 0.2573 | 0.8525 | 0.7604 | 0.8307 | 0.8898 | 0.8258 |
| scIB Metrics | scGraph-LCA | scGraph-RWR | ||||||||
| Method | NMI | ARI | ASW | Avg Bio | Spearman | Kendall’s | Pearson | Spearman | Kendall’s | Pearson |
| Ours | 0.7804 | 0.3431 | 0.5032 | 0.5422 | 0.5114 | 0.3887 | 0.5611 | 0.0937 | 0.0569 | 0.4427 |
| Geneformer | 0.7213 | 0.2241 | 0.4717 | 0.4724 | 0.2962 | 0.2157 | 0.3278 | 0.1369 | 0.0916 | 0.3605 |
| scGPT | 0.7740 | 0.3014 | 0.5074 | 0.5276 | 0.1339 | 0.0962 | 0.2058 | 0.0682 | 0.0447 | 0.3638 |
| CellPLM | 0.7915 | 0.3334 | 0.5243 | 0.5498 | 0.0980 | 0.0687 | 0.1933 | 0.1436 | 0.0962 | 0.3980 |
| scCello | 0.7693 | 0.3092 | 0.5199 | 0.5328 | 0.2417 | 0.1751 | 0.3384 | 0.0670 | 0.0427 | 0.4768 |
| Model | F1 Micro | F1 Macro | AP Micro | AP Macro |
| Ours | 0.783 0.01 | 0.632 0.02 | 0.832 0.01 | 0.632 0.02 |
| Geneformer | 0.767 0.01 | 0.593 0.02 | 0.832 0.01 | 0.613 0.02 |
| scCello | 0.781 0.01 | 0.627 0.02 | 0.823 0.01 | 0.620 0.02 |
| scGPT | 0.809 0.01 | 0.683 0.02 | 0.861 0.01 | 0.678 0.02 |
| Method | Pearson | Spearman | Kendall’s |
| Ours | |||
| Geneformer | |||
| CellPLM | |||
| scCello | |||
| scGPT |
| Model | MLM | DynamicMargin | OntoAlign | scIB | scGraph-LCA |
| MLM only | ✓ | ✗ | ✗ | 0.7597 | 0.4237 |
| w/o OntoAlign | ✓ | ✓ | ✗ | 0.8066 | 0.3607 |
| w/o DynamicMargin | ✓ | ✗ | ✓ | 0.7841 | 0.7211 |
| scOntoFM | ✓ | ✓ | ✓ | 0.7958 | 0.6112 |
| scIB | scGraph-LCA | |||||||
| Dataset | scOntoFM | w/o OntoAlign | w/o DynamicMargin | MLM only | scOntoFM | w/o OntoAlign | w/o DynamicMargin | MLM only |
| HIHA | 0.7533 | 0.7745 | 0.7848 | 0.7403 | 0.7157 | 0.4282 | 0.8301 | 0.4787 |
| Heart | 0.8324 | 0.8549 | 0.8480 | 0.7908 | 0.7074 | 0.4171 | 0.6824 | 0.5511 |
| Pancreas | 0.8140 | 0.8388 | 0.7767 | 0.7191 | 0.3992 | 0.3957 | 0.4408 | 0.3351 |
| Lung | 0.8467 | 0.8332 | 0.8119 | 0.8091 | 0.6777 | 0.2689 | 0.8622 | 0.4425 |
| Liver | 0.7730 | 0.7773 | 0.7837 | 0.7553 | 0.5704 | 0.3691 | 0.6911 | 0.4847 |
| Dataset | Reference | Target | # Ref Labels | # Target Labels | Agreement Score |
| HIHA | cell_type | AIFI_L1 | 38 | 9 | 0.259 |
| HIHA | cell_type | AIFI_L2 | 38 | 29 | 0.970 |
| HIHA | cell_type | AIFI_L3 | 38 | 71 | 0.830 |
| Kidney | cell_type | SubclassLevel1 | 33 | 17 | 0.933 |
| Kidney | cell_type | SubclassLevel2 | 33 | 97 | 0.407 |
| Noise | Accuracy (mean std) | Macro F1 (mean std) | Parent (mean std) | Child (mean std) | Sibling (mean std) |
| 0.1 | 0.941 0.001 | 0.806 0.017 | 0.001 0.000 | 0.003 0.001 | 0.014 0.000 |
| 0.2 | 0.940 0.001 | 0.752 0.013 | 0.001 0.000 | 0.003 0.001 | 0.015 0.000 |
| 0.3 | 0.934 0.002 | 0.691 0.008 | 0.001 0.000 | 0.005 0.001 | 0.015 0.001 |
| 0.4 | 0.834 0.036 | 0.572 0.024 | 0.001 0.001 | 0.053 0.021 | 0.018 0.010 |
| 0.5 | 0.150 0.018 | 0.207 0.025 | 0.000 0.000 | 0.552 0.002 | 0.035 0.003 |
Hierarchical Representation Learning
Hierarchical representation learning organizes data into nested, coarse-to-fine features that capture parts, patterns, and global structure.
See all 240 papers