MUSS: A Multi-scale and Sequence-based Model for Single-cell Gene Regulation

Kangjie Zheng; Amirhossein Vahidi; Liying Jin; Joseph Clarke; Vijaya Baskar MS; Connor Rogerson; Mengji Zhang; Muzlifah Haniffa

cs.AINov 5, 2026

MUSS: A Multi-scale and Sequence-based Model for Single-cell Gene Regulation

Authors: Kangjie Zheng, Amirhossein Vahidi, Liying Jin, Joseph Clarke, Vijaya Baskar MS, Connor Rogerson, Mengji Zhang, Muzlifah Haniffa,

Organizations: Wellcome Sanger Institute, Wellcome Genome Campus, Cambridge, UK · Cambridge Centre for AI in Medicine, University of Cambridge, Cambridge, UK · Cambridge University Hospitals Foundation Trust, Addenbrooke’s Hospital, Cambridge, UK · Cambridge Institute for Therapeutic Immunology and Infectious Diseases (CIITID), University of Cambridge, UK ·

Abstract

Modeling gene regulation in single cells requires jointly representing DNA sequence, chromatin accessibility, and gene expression across regulatory scales. Sequence-based models capture DNA-level regulatory grammar but often lack cell-level resolution, whereas many single-cell omics models represent genes and peaks as fixed-vocabulary tokens without their underlying sequence. Here, we present MUSS, a sequence-based multi-scale model for single-cell gene regulation that preserves both DNA-level and cell-level resolution. MUSS derives peak- and gene-level representations from DNA using a frozen pre-trained sequence encoder and integrates them through peak-peak self-attention and distance-biased peak-gene interactions. Peak masked language modeling followed by training on paired scATAC-seq and scRNA-seq aligns these representations with cell-specific regulatory states. Across gene expression prediction, enhancer-gene linking, motif-guided regulatory prioritization, and zero-shot cell clustering, MUSS achieves strong performance against sequence-based and single-cell omics baselines. It further predicts expression for genes and peaks unseen during task training. These results support sequence-grounded, multi-scale representations for decoding cell-type-specific regulatory programs. Code, datasets, and model checkpoints are available at https://github.com/Lotfollahi-lab/MUSS_neurips26.

Figures & tables

Figure 1: Muss constructs unified sequence-based representations for chromatin-accessible regions (peaks) and genes to capture global, multi-scale regulatory semantics. The illustration incorporates artwork created with BioRender.

Figure 2: Overview of our proposed Muss . Muss first constructs sequence-based embeddings for peaks and genes using a frozen AlphaGenome encoder. It then models regulatory dependencies through a multi-scale encoder, including peak–peak self-attention and bidirectional cross-scale attention. Finally, Muss is trained with peak masked language modeling and gene expression prediction to learn peak- and gene-level regulatory semantics.

Unseen sequence
		PBMC10K [ 48 ]			BMMC [ 29 ]			Openproblem [ 7 ]			Bone [ 39 ]
	Model	PCC $\uparrow$	CCC $\uparrow$	$R^{2}\uparrow$	PCC $\uparrow$	CCC $\uparrow$	$R^{2}\uparrow$	PCC $\uparrow$	CCC $\uparrow$	$R^{2}\uparrow$	PCC $\uparrow$	CCC $\uparrow$	$R^{2}\uparrow$
Seen cell types.
Seq	Borzoi [ 28 ]	0.560	0.237	-0.293	0.539	0.424	0.025	0.613	0.486	0.242	0.487	0.313	-0.040
	AlphaGenome [ 2 ]	0.609	0.402	-0.005	0.650	0.450	0.144	0.774	0.690	0.522	0.615	0.545	0.262
	scooby [ 22 ]	0.533	0.353	-0.088	0.655	0.543	0.293	0.709	0.696	0.312	0.655	0.648	0.278

Table 1: Gene expression prediction from scATAC-seq to scRNA-seq under four evaluation settings defined by sequence split ( seen seq vs. unseen seq ) and cell-type split ( seen cell types vs. unseen cell types ). Higher is better for PCC, CCC, and

R^{2}

. Metrics are computed on per-cell-type pseudobulk. Best and second-best values are bold and underlined, respectively.

Figure 3: Distance-stratified AUROC on the balanced GeneHancer enhancer–gene linking benchmark. Left: eQTL-supported pairs; right: CHi-C-supported pairs. Error bars are 95% bootstrap confidence intervals over 1,000 resamples; the dashed line denotes chance.

Model type	Model	ARI $\uparrow$	FMI $\uparrow$	NMI $\uparrow$	AMI $\uparrow$
Bulk	AlphaGenome [ 2 ]	0.1032 $\pm$ 0.0000	0.3814 $\pm$ 0.0000	0.1646 $\pm$ 0.0000	0.1633 $\pm$ 0.0000
	Borzoi [ 28 ]	0.0837 $\pm$ 0.0010	0.3577 $\pm$ 0.0013	0.1450 $\pm$ 0.0006	0.1437 $\pm$ 0.0006
	EpiBERT [ 23 ]	0.0909 $\pm$ 0.0002	0.3642 $\pm$ 0.0002	0.1499 $\pm$ 0.0008	0.1486 $\pm$ 0.0008
	GET [ 16 ]	0.0916 $\pm$ 0.0002	0.3671 $\pm$ 0.0004	0.1486 $\pm$ 0.0001	0.1473 $\pm$ 0.0001
Single cell	PeakVI [ 1 ]	0.2915 $\pm$ 0.0005	0.5041 $\pm$ 0.0004	0.4623 $\pm$ 0.0004	0.4615 $\pm$ 0.0005
	PoissonVI [ 30 ]	0.4103 $\pm$ 0.0045	0.5994 $\pm$ 0.0028	0.5656 $\pm$ 0.0035	0.5649 $\pm$ 0.0035

Table 2: Cell clustering results on PBMC169K [ 13 ] BIO_ddseq_1. Higher is better for all metrics. Values are mean

\pm

standard deviation over 10 clustering seeds.

Type	Ablation	Unseen sequence			Seen sequence
		PCC $\uparrow$	CCC $\uparrow$	$R^{2}$ $\uparrow$	PCC $\uparrow$	CCC $\uparrow$	$R^{2}$ $\uparrow$
Arch	w/o Sequence embedding	0.196 (-0.549)	0.132 (-0.600)	-0.330 (-0.837)	0.958 (-0.011)	0.947 (-0.016)	0.890 (-0.036)
Arch	w/o Cross-scale Interact	0.009 (-0.736)	0.002 (-0.730)	-1.422 (-1.929)	0.967 (-0.002)	0.959 (-0.004)	0.920 (-0.006)
Pos. Info	w/o Distance Reweighting	0.736 (-0.009)	0.722 (-0.010)	0.480 (-0.027)	0.967 (-0.002)	0.958 (-0.005)	0.915 (-0.011)
	w/o Peak RoPE	0.725 (-0.020)	0.717 (-0.015)	0.454 (-0.053)	0.959 (-0.010)	0.953 (-0.010)	0.905 (-0.021)
	w/o RBF distance	0.707 (-0.038)	0.700 (-0.032)	0.426 (-0.081)	0.959 (-0.010)	0.953 (-0.010)	0.904 (-0.022)

Table 3: BMMC gene expression ablations. Differences from the full Muss model are shown in parentheses.

Dataset	Sequence Encoder	Unseen sequence			Seen sequence
		PCC $\uparrow$	CCC $\uparrow$	$R^{2}$ $\uparrow$	PCC $\uparrow$	CCC $\uparrow$	$R^{2}$ $\uparrow$
PBMC10K	Borzoi	0.698	0.655	0.439	0.914	0.885	0.798
	NT-v3	0.459	0.409	0.083	0.912	0.887	0.801
	AlphaGenome	0.684	0.663	0.409	0.909	0.895	0.811
BMMC	Borzoi	0.695	0.692	0.430	0.986	0.984	0.969
	NT-v3	0.452	0.437	0.037	0.983	0.980	0.961

Table 4: Gene-expression prediction with different frozen DNA sequence encoders on PBMC10K, BMMC, and Openproblem (OP). Bold marks the best result per metric within each dataset and split, including ties.

Figure 4: Few-shot scaling curves for gene expression prediction on PBMC10K. The two panels show the PCC and CCC of Muss as the number of paired cells used for fine-tuning increases.

Figure 5: DNA sequence motif–informed peak–gene prioritization across genomic distances. Left: top-10 enrichment of known TF–gene links among top-ranked peaks (averaged per gene). Right: pairwise AUROC comparing positive peaks (broad and conservative definitions) to distance-matched controls. Error bars show 95% CI; dashed line indicates chance.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Training objective	Peak MLM
Dataset	PBMC169K
Input sequence embedding	AlphaGenome embedding
Input dimension	3072
Model dimension	2048
Encoder layer types	1 layer of peak self-attention, 2 layers of cross interaction blocks

Appendix

Table 5: Muss hyperparameters for peak MLM pre-training.

Dataset	Held-out cell types
PBMC10K	TEM/TEMRA cytotoxic T cells; regulatory T cells; classical monocytes;
	non-classical monocytes; DC2; pDC
BMMC	Erythroblast; proerythroblast; normoblast; CD16 + mono;
	CD8 + T naive; pDC; ID2-hi myeloid progenitor; plasma cell; ILC
Openproblem	Erythrocyte progenitor; megakaryocyte progenitor;
	monocyte progenitor; B-cell progenitor

Appendix

Table 6: Held-out cell types used in the unseen-cell-type setting.

Hyperparameter	Value
Initialization	Peak MLM checkpoint
Input sequence embedding	AlphaGenome embedding
Input dimension	3072
Model dimension	2048
Encoder layer types	1 layer of peak self-attention, 2 layers of cross interaction blocks
Number of attention heads	16

Appendix

Table 7: Muss hyperparameters for gene expression prediction fine-tuning.

Model	ARI	FMI	NMI	AMI
AlphaGenome	$0.1032\pm 0.0000$	$0.3814\pm 0.0000$	$0.1646\pm 0.0000$	$0.1633\pm 0.0000$
ChromFound	$0.4835\pm 0.0082$	$0.6604\pm 0.0042$	$0.5950\pm 0.0036$	$0.5944\pm 0.0036$
PCA	$0.1649\pm 0.0221$	0.4473	0.2856	0.2845
Muss	$\mathbf{0.5377\pm 0.0008}$	$\mathbf{0.6973\pm 0.0006}$	$\mathbf{0.6085\pm 0.0003}$	$\mathbf{0.6079\pm 0.0003}$

Appendix

Table 8: PCA comparison for zero-shot cell clustering. Values are mean

\pm

standard deviation where reported; PCA standard deviations are available only for ARI.

Training cells $N$	Epochs	Total optimizer steps
1	200	200
2	200	400
4	200	800
8	150	1,200
16	100	1,600
32	80	2,560

Appendix

Table 9: Training schedule for the PBMC10K few-shot evaluation. Since the batch size is 1, the total number of optimizer steps is equal to

N\times

epochs.

Dataset	CT	EpiAgent	Muss
PBMC10K	Seen	0.763 / 0.524 / 0.092	0.909 / 0.895 / 0.811
	Unseen	0.746 / 0.503 / $-0.009$	0.893 / 0.887 / 0.788
BMMC	Seen	0.819 / 0.575 / 0.131	0.983 / 0.981 / 0.962
	Unseen	0.818 / 0.620 / 0.248	0.948 / 0.937 / 0.873
Openproblem	Seen	0.876 / 0.683 / 0.116	0.996 / 0.995 / 0.990
	Unseen	0.860 / 0.669 / 0.079	0.979 / 0.978 / 0.956

Appendix

Table 10: EpiAgent comparison on seen sequences. Entries are PCC / CCC /

R^{2}

; CT denotes the cell-type split.

Seen sequences
Model	PBMC10K	BMMC	Openproblem	Bone
CMAE	0.861	0.680	0.627	0.827
scMoGNN	0.903	0.720	0.689	0.805
ChromFound	0.837	0.658	0.534	0.774
Muss	0.876	0.732	0.691	0.811
Unseen sequences

Appendix

Table 11: Single-cell gene expression PCC. Bold denotes the highest value within each sequence setting.

	Seen sequences		Unseen sequences
Model	Seen CT	Unseen CT	Seen CT	Unseen CT
CMAE	$0.463\pm 0.038$	$0.458\pm 0.036$	—	—
scMoGNN	$0.859\pm 0.059$	$0.827\pm 0.049$	—	—
ChromFound	$0.856\pm 0.007$	$0.822\pm 0.007$	—	—
Borzoi	$0.719\pm 0.019$	$0.690\pm 0.016$	$0.567\pm 0.022$	$0.550\pm 0.016$
AlphaGenome	$0.696\pm 0.046$	$0.650\pm 0.058$	$0.656\pm 0.012$	$0.628\pm 0.020$

Appendix

Table 12: BMMC pseudobulk PCC (mean

\pm

standard deviation, five seeds). CT denotes cell types; dashes indicate unsupported unseen-sequence evaluation.

Dataset	Sequences	PCA+Ridge	Muss
PBMC10K	Seen	0.900 / 0.878 / 0.782	0.909 / 0.895 / 0.811
PBMC10K	Unseen	0.000 / 0.000 / $-1.310$	0.684 / 0.663 / 0.409
BMMC	Seen	0.972 / 0.963 / 0.931	0.983 / 0.981 / 0.962
BMMC	Unseen	0.000 / 0.000 / $-1.707$	0.748 / 0.738 / 0.524

Appendix

Table 13: PCA+Ridge comparison. Entries are PCC / CCC /

R^{2}

Dataset	CT	PCC	CCC	$R^{2}$
PBMC10K	Seen	-0.024	-0.023	-1.078
	Unseen	-0.013	-0.012	-1.115
BMMC	Seen	-0.029	-0.022	-0.789
	Unseen	-0.020	-0.015	-0.861
Openproblem	Seen	-0.026	-0.024	-0.665
	Unseen	-0.017	-0.016	-0.664

Appendix

Table 14: EpiAgent unseen-sequence failure diagnostics. CT denotes the cell-type split. These values do not represent native unseen-gene support.

Genomic context	Unseen seq.		Seen seq.
	PCC $\uparrow$	CCC $\uparrow$	PCC $\uparrow$	CCC $\uparrow$
Both sides ( Muss )	0.684	0.663	0.909	0.895
Upstream only	0.701	0.659	0.885	0.864

Appendix

Table 15: PBMC10K comparison of two-sided and upstream-only genomic context using AlphaGenome. Bold marks the better value per metric.

Model	ARI	FMI	NMI	AMI
AlphaGenome	$0.1032\pm 0.0000$	$0.3814\pm 0.0000$	$0.1646\pm 0.0000$	$0.1633\pm 0.0000$
Muss w/o peak MLM	$0.1221\pm 0.0006$	$0.3773\pm 0.0005$	$0.2068\pm 0.0007$	$0.2056\pm 0.0007$
Muss	$\mathbf{0.5377\pm 0.0008}$	$\mathbf{0.6973\pm 0.0006}$	$\mathbf{0.6085\pm 0.0003}$	$\mathbf{0.6079\pm 0.0003}$

Appendix

Table 16: Peak-MLM ablation for zero-shot cell clustering. Entries are means

\pm

standard deviations.

Model	AUROC (bal.)	AUPRC (bal.)	AUPRC micro	AUPRC macro	NDCG@10	HR@10
Muss	0.6590	0.6499	0.1120	0.2051	0.2095	0.5571
Muss w/o peak MLM	0.6106	0.6210	0.0422	0.1389	0.1479	0.3947
Difference	0.0484	0.0289	0.0698	0.0662	0.0616	0.1624

Appendix

Table 17: Peak-MLM ablation for eQTL enhancer–gene linking. “Bal.” denotes the balanced positive/negative evaluation. The last row is full minus ablated.

	AUPRC		NDCG			HR
Model	Micro	Macro	@5	@10	@20	@5	@10	@20
AlphaGenome	0.0718	0.1363	0.1233	0.1403	0.1694	0.2999	0.4114	0.5421
Borzoi	0.0834	0.1581	0.1452	0.1673	0.1998	0.3488	0.4702	0.6052
GET	0.0357	0.1014	0.0971	0.1072	0.1288	0.2813	0.3982	0.5377
EpiBERT	0.0445	0.1016	0.0996	0.1081	0.1317	0.2847	0.3958	0.5245
Muss	0.1271	0.2500	0.2331	0.2664	0.3157	0.4868	0.6492	0.7784

Appendix

Table 18: eQTL enhancer–gene retrieval within

\pm 1

Mb, at approximately 1:36 class imbalance. Higher is better.

Distance	Model	AUPRC micro	AUPRC macro	NDCG@10	HR@10
All	PCA+Ridge	0.0532	0.0920	0.1054	0.4056
	Muss	0.1271	0.2500	0.2664	0.6492
0–50 kb	PCA+Ridge	0.2008	0.3438	0.3955	0.8677
	Muss	0.2039	0.3672	0.3974	0.8478
50–100 kb	PCA+Ridge	0.1500	0.3872	0.4640	0.8690
	Muss	0.1875	0.4859	0.5535	0.8946

Appendix

Table 19: PCA+Ridge versus Muss for eQTL retrieval. Bold indicates the higher value within each distance bin.

Model	Mean (ms/cell)	Median (ms/cell)	Throughput (cells/s)
CMAE	0.99	0.99	1,006
ChromFound	5.94	5.93	168
Muss	61.18	55.07	16.3

Appendix

Table 20: Inference on 2,000 BMMC test cells using one H200. Timing and throughput are reported with the precision of the measurements.

Peaks per cell	8,192	16,384	24,576	32,768	49,152	65,536
GPU memory (GB)	29.2	40.4	55.0	69.5	98.7	127.8

Appendix

Table 21: Training memory for simulated cells with increasing peak counts. Settings: BF16, one cell per GPU, memory-efficient exact attention.

Genomics

Genomics uses computational and experimental methods to study genomes, genetic variation, gene regulation, and their links to traits and disease.

See all 132 papers

CellMSA: Context Modeling for Single-Cell Representation Learning
Sep 30, 2026Suyuan Zhao, Minghao Liu, Yizhen Luo +1Representation Learning Single-Cell Transcriptomics
Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · Tsinghua Institute of Multidisciplinary Biomedical Research (TIMBR), Tsinghua University +2
Prior-Guided Multi-Omic Transformers for Single-Cell Gene Regulatory Network Inference
May 30, 2026Tianyang Xu, Tianci Liu, Niraj Rayamajhi +4Transformer Gene Regulatory Network Inference
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA · Department of Horticulture and Landscape Architecture, Purdue University, West Lafayette, IN, USA · School of Biological Sciences, Illinois State University, Normal, IL, USA
Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT
Mar 3, 2026Ihor KendiukhovGenomic Foundation Models Single-Cell Transcriptomics
Department of Computer Science, University of Tübingen, Tübingen, Germany

Sep 30, 2026J/K move · Enter open · S save

CellMSA: Context Modeling for Single-Cell Representation Learning

Suyuan Zhao, Minghao Liu, Yizhen Luo, Zaiqing Nie

Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · Tsinghua Institute of Multidisciplinary Biomedical Research (TIMBR), Tsinghua University+2

Representation Learning Single-Cell Transcriptomics Single-Cell RNA Sequencing

Open paper AskPDF arXiv

Single-cell transcriptomics enables profiling of cellular states at unprecedented resolution, but its high dimensionality, sparsity, and technical batch effects pose significant challenges for representation learning. Existing single-cell foundation models typically encode each cell independently or only model cells from the same batch for denoising, thereby underutilizing the rich relational information across batches and cell types to model gene expression patterns. We argue that single-cell models can benefit from more informative cell-context modeling. By comparing consistency and variation across cells, models can capture fine-grained gene-gene dependencies associated with cell states, which are essential for learning high-quality representations. Inspired by the use of multiple sequence alignment (MSA) context in protein modeling, we propose CellMSA, a single-cell representation learning framework that introduces an MSA-inspired inductive bias into transcriptomic modeling. For each target cell, CellMSA retrieves relevant cells from different batches and biologically related cell types as context, and summarizes cross-cell patterns into a context-dependent gene-pair representation. This representation is then injected into a pair-aware target-cell encoder for fine-grained representation learning. We pretrain CellMSA on a large-scale human single-cell corpus of approximately 109 million cell observations, including 65.6 million primary observations. Experiments show that our framework consistently outperforms existing methods across multiple benchmarks. Code is available at the following repository: https://github.com/PharMolix/CellMSA.

Abstract

Genomics

Explore similar work

CellMSA: Context Modeling for Single-Cell Representation Learning

Prior-Guided Multi-Omic Transformers for Single-Cell Gene Regulatory Network Inference

Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT