cs.LGFeb 16, 2026

Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics

Authors: Ihor Kendiukhov

Organizations: Department of Computer Science, University of Tübingen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored. We present the first systematic study of scaling behaviour for masked-reconstruction transformers trained on single-cell RNA sequencing (scRNA-seq) data. Using expression profiles from the CELLxGENE Census, we construct two experimental regimes: a data-rich regime (512 highly variable genes, 200,000 cells) and a data-limited regime (1,024 genes, 10,000 cells). Across seven model sizes spanning three orders of magnitude in parameter count (533 to 3.4 x 10^8 parameters), we fit the parametric scaling law to validation mean squared error (MSE). The data-rich regime exhibits clear power-law scaling with an irreducible loss floor of c ~ 1.44, while the data-limited regime shows negligible scaling, indicating that model capacity is not the binding constraint when data are scarce. These results establish that scaling laws analogous to those observed in natural language processing do emerge in single-cell transcriptomics when sufficient data are available, and they identify the data-to-parameter ratio as a critical determinant of scaling behaviour. A preliminary conversion of the data-rich asymptotic floor to information-theoretic units yields an estimate of approximately 2.30 bits of entropy per masked gene position. We discuss implications for the design of single-cell foundation models and outline the additional measurements needed to refine this entropy estimate.

Figures & tables

SizeddLLHHFFN multParams ( V ⁣= ⁣512V\!=\!512 )
XXS1111533
TINY161119,953
XS64244132,993
S128484859,137
M51268419,178,497
L10208124100,510,801
Table 1: Model size presets used in the standardized V ⁣= ⁣512V\!=\!512 experiments. dd : model dimension; LL : number of encoder layers; HH : number of attention heads; FFN mult: feed-forward expansion factor.
SizeParamsVal. MSETest MSE
XXS5332.101±0.0322.101\pm 0.0322.110±0.0332.110\pm 0.033
TINY9,9531.600±0.0101.600\pm 0.0101.617±0.0071.617\pm 0.007
XS132,9931.515±0.0111.515\pm 0.0111.533±0.0071.533\pm 0.007
S859,1371.482±0.0081.482\pm 0.0081.500±0.0071.500\pm 0.007
M19,178,4971.462±0.0031.462\pm 0.0031.479±0.0041.479\pm 0.004
L100,510,8011.477±0.0091.477\pm 0.0091.494±0.0111.494\pm 0.011
Table 2: Standardized V ⁣= ⁣512V\!=\!512 summary (200k cells, 60k steps, 3 seeds per size). Mean ±\pm standard deviation is reported across seeds for both validation and held-out test MSE.
Splitnnα\alphaaaccR2R^{2}
Validation180.2662.1531.4440.858
Test180.2682.1681.4620.859
Table 3: Power-law fits for the standardized V ⁣= ⁣512V\!=\!512 sweep (200k cells, 18 runs). Reporting both validation and held-out test fits confirms that the qualitative conclusion does not depend on checkpoint selection by validation alone.
Figure 1: Standardized V ⁣= ⁣512V\!=\!512 scaling on log-log axes. Each point is one training run (best checkpoint per seed). The held-out test fit closely matches the validation fit, with α≈0.267\alpha\approx 0.267 and R2≈0.859R^{2}\approx 0.859 on both splits.
Figure 2: Additional analyses at V ⁣= ⁣512V\!=\!512 . Left: at fixed 60k steps, the M-model data-size sweep is nearly flat in validation MSE ( β≈−0.002\beta\approx-0.002 , R2=0.61R^{2}=0.61 ), indicating that a clean L(D)L(D) law still requires matched optimization budgets. Right: residual error is strongly gene-dependent in a representative held-out L run, with correlation 0.892 between mean expression and per-gene MSE.
Metricnnα\alphaaaccR2R^{2}Entropy floor
      (bits/masked pos.)
MSE180.2662.1531.4440.8582.312
Gauss. NLL180.1820.4011.5920.8382.296
Table 4: Scaling-law fits for the standardized V ⁣= ⁣512V\!=\!512 sweep (200k cells, 18 runs), with entropy floor estimates. The NLL values are derived from MSE under a Gaussian assumption (Equation 4 ), so the two estimates are not fully independent; Figure 2 shows that residual variance is in fact gene-dependent.
Figure 3: Standardized V ⁣= ⁣512V\!=\!512 scaling fits (18 runs, 6 sizes ×\times 3 seeds, all 60k steps). Left: MSE metric. Right: derived Gaussian NLL metric. Under the Gaussian homoscedastic approximation, both correspond to about 2.30 bits of irreducible uncertainty remaining per masked position.
More in this topic

Neural Scaling Laws

Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.

See all 79 papers

Explore similar work

CardsList
Aug 3, 2026

Scaling an Autoregressive Transformer for Single-Cell Generation

We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.

Aleksandr Sharipov, Yusif Mukhtarov, Igor Molybog

HawAII, University of Hawai’i at Manoa

TransformerAutoregressive GenerationSingle-Cell TranscriptomicsAI Scaling Laws
AskPDF
Jun 8, 2026

Integrating gene regulatory priors into Transformer attention with scTransformer for interpretable scRNA-seq analysis

Motivation: Transformer-based models are increasingly applied to large-scale single-cell transcriptomics, showing strong performance through self-supervised learning on millions of cells. However, most existing approaches treat genes as independent features, and largely ignore prior biological knowledge, which limits interpretability and robustness. In this paper, we explore whether explicitly incorporating gene regulatory information can improve both model performance and biological insight. Results: We present scTransformer, the first Transformer-based approach that builds a priori knowledge of biological mechanisms into the model's attention patterns. By constraining information flow according to known regulatory structures, the model learns representations that are more biologically meaningful. We evaluate scTransformer on a disease-relevant single-nucleus RNA-seq dataset using supervised cell-type classification. Compared to standard Transformers, our approach improves classification accuracy, enhances separation of cell types in embedding space, and produces attention patterns consistent with known regulatory programs. Overall, our results demonstrate that embedding biological structure into Transformer models can enhance interpretability without sacrificing performance, offering a principled step toward biologically grounded foundation models for single-cell omics.

Mikele Milia, Louis Fabrice Tshimanga, Henning Mueller +2

Department of Information Engineering, University of Padova, Padova, Italy · Department of Neuroscience, University of Padova, Padova, Italy · Padova Neuroscience Center, Padova, Italy +3

Transformer InterpretabilityTransformer AttentionScientific MLGenomicsSingle-Cell RNA Sequencing
AskPDF
Feb 18, 2026

Parameter-free representations outperform single-cell foundation models on downstream benchmarks

Single-cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use transformer-based architectures to learn a generative model for gene expression by embedding genes into a latent vector space. These embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple benchmarks commonly used to evaluate single-cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of cell identity can be captured by simple linear representations of single cell gene expression data.

Huan Souza, Pankaj Mehta

Department of Physics, Boston University, Boston, MA, 02215, USA · Faculty of Computing and Data Science, Boston University, Boston, MA, 02215, USA

Single-Cell RNA SequencingLinear Representation HypothesisOOD Generalization
AskPDF