Organizations: Department of Pathology and Laboratory Medicine, University of Pennsylvania, Philadelphia, PA, USA · Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA · Germantown Friends School, Philadelphia, PA, USA · Department of Biostatistics, Epidemiology & Informatics, University of Pennsylvania, Philadelphia, PA, USA
Identifying cell types directly from routine haematoxylin and eosin (H&E) histology would enable single-cell analysis at scale, but training such models has relied on manual pathologist annotations, which are slow, expensive and unreliable for many cell types. We instead supervise morphology with molecules. Imaging-based spatial transcriptomics profiles individual cells in situ on a section that can afterwards be stained with H&E, so that molecular identity and morphology are observed for the same physical cell. We assembled 81 such paired Xenium sections spanning 16 organs, derived per-cell labels by clustering, marker-gene annotation, organ-wise human review and quality control, and mapped them onto the cell types commonly reported in each organ. This yielded 15.4 million cells, each with a paired H&E image patch and one of 23 cell types, on which we trained CytoFormer, a cell foundation model with a multi-task, per-organ classification head. On spatially held-out tissue CytoFormer reached an accuracy of 0.85 and a macro-F1 of 0.78 across all 16 organs, and its predictions reproduced the tissue architecture of an entire held-out section. The representation also transfers: with the encoder frozen, a linear head on CytoFormer features performed better than six pathology foundation models on four expert-annotated benchmarks, including on organs and cell types that were not part of pretraining. Finally, in an interactive active-learning setting, CytoFormer's embeddings are markedly more label-efficient than existing pathology foundation models, detecting normal epithelium amid look-alike tumour with an F1 of 0.82 from only a few annotations and leading the strongest baseline by 0.13 in F1. CytoFormer turns paired H&E and spatial transcriptomics into a reusable, label-efficient representation for cell-level analysis of routine histology.
Figures & tables
Figure 1
Figure 2: CytoFormer classifies cell types on held-out H&E tissue across 16 organs. (a) F1 per cell type and (b) macro-F1 per organ on the spatially held-out test split. (c–e) A breast section held out in its entirety: (c) raw H&E, (d) the cell-type reference derived from the paired spatial transcriptomics and (e) the CytoFormer prediction. (f) Confusion matrix for that section. (g) Crop field-of-view ablation: one fixed set of cells is cropped at each field of view and resized to 224×224 pixels. (h) Macro-F1 and (i) accuracy at each crop field of view. Error bars represent 95 % confidence intervals from bootstrap resampling over test cells. Statistical significance against the 56μ m setting was assessed using a paired two-sided bootstrap test: ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 . A bracket is drawn only between the 56μ m setting and the second-best setting and gives their exact p -value; the other settings are marked with asterisks only.
Figure 3Figure 4Figure 5Figure 6
Extended Data Figure 2 : Epithelium detection F1 across active-learning iterations. Epithelium vs all others; 10,627 positive / 151,075 negative cells; same annotations, schedule and 5 seeds for every model. F1 for the Epithelium class, mean of 5 seeds. Bold = best in row.
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4