Rethinking Class Imbalance for Single-Cell Foundation Models: A Systematic Benchmark Across Architectures and Long-Tail Loss Functions

Zeyu Dong; Jiahui Zhong

cs.LGSep 20, 2026

Rethinking Class Imbalance for Single-Cell Foundation Models: A Systematic Benchmark Across Architectures and Long-Tail Loss Functions

Authors: Zeyu Dong, Jiahui Zhong

Abstract

Single-cell foundation models (scGPT, scBERT, Geneformer) achieve cell-type classification accuracy up to 97.5% in our experiments, yet this aggregate accuracy can mask systematic failure on rare, often disease-relevant cell populations that long-tail loss functions are widely assumed to address. We present a systematic benchmark of six long-tail loss functions (cross-entropy, weighted CE, class-balanced loss, focal loss, LDAM, logit-adjusted softmax) across three architectures and three datasets (Multiple Sclerosis, Zheng68K, human Pancreas), totaling 162 controlled training runs (3 backbones x 3 datasets x 6 losses x 3 seeds). The gap between overall accuracy, Macro-F1, and rare-class recall under plain cross-entropy is consistent across all nine (architecture, dataset) settings, driven by dataset structure rather than pretraining. Rare-class failure itself splits into two regimes with distinct embedding-geometry signatures, visible before any loss is chosen: some classes are recoverable by the right loss, while others retain linear separability yet are absorbed into unrelated classes' neighborhoods under every evaluated loss and architecture. Among the recoverable classes, the efficacy of reweighting is predicted by a class's absolute training-set size, rather than its share of the dataset or the dataset's overall imbalance ratio. Class-balanced loss and LDAM are the most consistent choices across all nine settings, while logit adjustment trades rare-class precision for recall rather than improving both. Our results give both a reusable benchmark and mechanism-grounded practical guidelines for combining foundation models with imbalanced biological data.

Figures & tables

Figure 1: Training-set class rank vs. count for all three datasets (log scale). MS’s and Pancreas’s smallest classes reach a minimum of single-digit sample counts, while Zheng68K’s smallest class still has

\sim

70 training cells.

Figure 2: Left: per-dataset breakdown under plain CE (mean of 9 backbone

\times

seed runs per bar); Pancreas has the highest accuracy but also the largest accuracy-to-rare-recall drop. Right: rare-class precision vs. recall by loss (Section 4.4 ); dashed line is

y=x

. Points above trade recall for precision, points below trade precision for recall.

Loss	F1	$\text{Recall}_{\mathcal{R}}$	F1	$\text{Recall}_{\mathcal{R}}$	F1	$\text{Recall}_{\mathcal{R}}$	$\Delta$ F1	Rare AUPRC	Mean Rank
	MS		Zheng68K		Pancreas
Cross-entropy	0.656 ±.054	0.596 ±.072	0.711 ±.026	0.610 ±.054	0.794 ±.052	0.710 ±.106	–	0.712 ±.083	4.56
Weighted CE	0.721 ±.035	0.687 ±.042	0.704 ±.041	0.634 ±.050	0.816 ±.039	0.772 ±.065	$+0.026$	0.728 ±.066	2.78
Class-balanced	0.716 ±.042	0.685 ±.047	0.708 ±.042	0.649 ±.050	0.816 ±.028	0.746 ±.068	$+0.026$	0.734 ±.062	2.44
Focal	0.662 ±.059	0.605 ±.071	0.718 ±.027	0.625 ±.042	0.787 ±.034	0.704 ±.081	$+0.002$	0.719 ±.068	3.44
LDAM	0.693 ±.050	0.644 ±.066	0.717 ±.031	0.625 ±.048	0.819 ±.021	0.767 ±.047	$+0.022$	0.690 ±.095	2.56

Table 1: Macro-F1 and rare-class recall (

n{=}9

: 3 backbones

\times

3 seeds;

\pm

is the standard deviation across these nine cells) per loss and dataset, plus mean

\Delta

F1 over CE (Eq. 3 ), rare-class macro-AUPRC (area under the precision-recall curve per rare class, macro-averaged, mean over the same nine cells;

n{=}24

for cross-entropy,

n{=}27

for every other loss), and mean rank (1=best, 6=worst) across all nine cells. Bold marks the best value per column.

Panel of Figure 3 — Figure 3: Left: per-class F1 heatmap (CE, MS), rows sorted by mean F1 across the three backbones, hardest (top) to easiest (bottom); a class with high F1 on one backbone but low F1 on the other two can still rank as hard. * marks a rare class (Section 3 ). The two hardest classes are dark across all three columns since no backbone architecture recovers them; classes further down show lighter, more backbone-dependent shading (three-way split, Section 5 ). Right: per-class $\Delta$ F1 against training-set size (log $x$ -axis) for all rare classes across all three datasets; color marks dataset. Circles (fit line, $n{=}19$ , $r{=}{-}0.77$ ) are the recoverable regime; $\times$ markers show the four severely entangled classes (two from MS, one each from Zheng68K and Pancreas); the same near-floor regime recurs across all three datasets (Section 4.3 ).

Figure 4: UMAP ( Healy and McInnes 2024 ) of MS test-set embeddings (scGPT), with the two severely entangled classes’ 116 test-set cells (oligodendrocyte C: 34, phagocyte: 82) highlighted against all other classes in gray throughout all three panels. Left: true labels in CE’s embedding space; these 116 cells appear in orange. Center: the same 116 cells in the same CE embedding space, now colored by CE’s prediction correctness (blue = correctly classified, orange = misclassified). Right: the same 116 cells in LDAM’s own, separately fine-tuned embedding space, colored by LDAM’s prediction correctness, since CE and LDAM are separately fine-tuned checkpoints. Both losses recover only a small minority of these cells (9/116 under CE, 7/116 under LDAM, both blue), confirming the failure is rooted in the embedding geometry itself, not a specific classifier head.

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 256 papers

Feb 18, 2026

Parameter-free representations outperform single-cell foundation models on downstream benchmarks

Single-cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use transformer-based architectures to learn a generative model for gene expression by embedding genes into a latent vector space. These embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple benchmarks commonly used to evaluate single-cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of cell identity can be captured by simple linear representations of single cell gene expression data.

Huan Souza, Pankaj Mehta

Department of Physics, Boston University, Boston, MA, 02215, USA · Faculty of Computing and Data Science, Boston University, Boston, MA, 02215, USA

Single-Cell RNA Sequencing Linear Representation Hypothesis OOD Generalization

Ask PDF

Jul 10, 2026

A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition

Long-tailed recognition methods often modify losses, margins, or representations to reduce the dominance of frequent classes. We ask whether, after Balanced Softmax training, the remaining tail error can be reduced by retraining only the classifier. We evaluate BS-cRT, a two-stage procedure that trains a backbone and cosine classifier with Balanced Softmax, freezes the backbone, and updates only the classifier on balanced episodic batches. The second stage keeps the empirical-prior Balanced Softmax objective and uses raw cosine logits at inference. Across CIFAR-100-LT, CIFAR-10-LT, ImageNet-LT, and Places-LT, this classifier-only step consistently improves Few-shot accuracy over the matched Balanced Softmax checkpoint. At imbalance factor 100, Few-shot gains are +5.15 points on CIFAR-100-LT and +5.83 on CIFAR-10-LT; on ImageNet-LT and Places-LT, gains are +6.92 and +9.78 points, respectively, with a Top-1/Few-shot trade-off on ImageNet-LT. We also analyze Counterfactual Boundary Risk Minimization (CBRM), a boundary-probe extension using prototype-based features near decision boundaries. CBRM identifies two failure modes: scaled-logit cosine margins destabilize training, and corrected hardest-negative probes remain head-class anchored. The results support BS-cRT as a practical classifier-side baseline and indicate that boundary supervision must account for class frequency.

Juan Terven, Diana Margarita Córdova Esparza, Julio Alejandro Romero Gonzalez +4

CICATA Querétaro Instituto Politecnico Nacional Mexico · Facultad de Informática Universidad Autónoma de Querétaro Mexico

Class-Imbalanced Learning Long-Tailed Classification

Ask PDF

Jul 6, 2026

Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap

Most classification problems assume the classes are roughly separable, so that an individual sample can usually be assigned to one class. Single-cell perturbation data violates this assumption: two perturbations can produce different populations of cells while overlapping so much that an individual cell could belong to either. Per-cell accuracy then measures this overlap rather than model quality. We see this on Tahoe-100M and the Virtual Cell Challenge, where a linear classifier, an MLP, and a Transformer all plateau near macro-F1 0.2-0.3 even though almost every pair of perturbations is statistically distinguishable. The fix is to score perturbations across the whole population rather than cell by cell. We average a classifier's per-cell probability vectors over all cells of a perturbation to form a population profile, then rank candidate perturbations by this profile; we call the resulting score the Classifier Discrimination Score (CDS). Taking the top-ranked class recovers the winning perturbation. It needs no retraining, costs linear time in the number of cells, and recovers near-perfect identification from the same weak models. CDS differs from the pseudobulk-based Perturbation Discrimination Score (PDS) used in recent benchmarks only in where the average is taken, raw gene expression for PDS versus a learned discriminative space for CDS, and identifies the true perturbation more reliably on both datasets, with the gap widening as cells grow scarce. Because a metric that misranks the ground truth will misrank the models scored against it, per-cell accuracy and raw-pseudobulk scores should be used with caution when comparing perturbation models.

Youssef Marrakchi, Davide D'Ascenzo, Sebastiano Cultrera di Montesano

Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA. · Department of Computer Science, University of Milan, Milan, Italy. · Department of Control and Computer Engineering, Politecnico di Torino, Torino, Italy. +1

Single-Cell Perturbation Modeling Classification Single-Cell RNA Sequencing

Ask PDF

Abstract

Class-Imbalanced Learning

Explore similar work

Parameter-free representations outperform single-cell foundation models on downstream benchmarks

A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition

Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap