Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching

Rasched Haidari; Sam Martin; Maxime Allard

cs.LGMar 27, 2026

Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching

Authors: Rasched Haidari, Sam Martin, Maxime Allard

Organizations: Helical London, UK

Abstract

Large Genomic Foundation Models have recently achieved remarkable results and in-vivo translation capabilities. However these models quickly grow to over a few Billion of parameters and are expensive to run when compute is limited. To overcome this challenge, we present a distillation framework for transferring mRNA representations from a state of the art genomic foundation model into a much smaller model specialized for mRNA sequences, reducing the size by 200-fold. Embedding-level distillation worked better than logit based methods, which we found unstable. Benchmarking on mRNA-bench demonstrates that the distilled model achieves state-of-the-art performance among models of comparable size and competes with larger architectures for mRNA-related tasks. Our results highlight embedding-based distillation of mRNA sequences as an effective training strategy for biological foundation models. This enables similar efficient and scalable sequence modelling in genomics, particularly when large models are computationally challenging or infeasible.

Figures & tables

Figure 1: The student model is aligned with two hidden layers (5th and 8th) using projections from layers of Evo2-1B (12th and 25th) .

Panel of Figure 2 — Figure 2: Benchmark results.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure A1: Losses for the train and validation set

Panel of Figure 4 — Figure A2: Student model gradient norms and embedding variances during training

Panel of Figure 5 — Figure A3: Cosine training loss for the first and second matched layers.

Panel of Figure 6 — Figure A4: MSE Losses for training and validation data

Figure A5: Central Kernel Alignment (CKA) Values Pre- and Post- Linear Projection Layer during training.

Figure A6: Entropy against position from the start of the mRNA sequence. Abrupt changes in the logit distribution and regions of high entropy make learning difficult.

Variance Threshold	block 12	norm
50%	1	2
75%	1	4
90%	1	5
95%	1	6
99%	431	6

Appendix

Table 1: Number of principal components required to reach different variance thresholds for Evo2 hidden layers.

Figure A7: Summary entropy statistics across 100 mRNA sequences. The average probability per token is slightly better than uniform.

Figure A8: UMAP over Phylum for Evo2 and the student model.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

Reverse Distillation: Consistently Scaling Protein Language Model Representations
Mar 8, 2026Darius Catrina, Christian Bepler, Samuel Sledzieski +1Protein Language Models Protein Representation Learning
Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
Jun 3, 2026Daria Ledneva, Denis KuznetsovAdaptive Tokenization Genomic Foundation Models
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia.
RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling
Aug 24, 2026Ziyuan Wang, Bohao Tang, Fei Zhang +2Long-Context Modeling Molecular Representation Learning
Shanghai Innovation Institute · Center for Excellence in Molecular Cell Science, CAS · University of Chinese Academy of Sciences +2

Mar 8, 2026J/K move · Enter open · S save

Reverse Distillation: Consistently Scaling Protein Language Model Representations

Darius Catrina, Christian Bepler, Samuel Sledzieski, Rohit Singh

Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA

Protein Language Models Protein Representation Learning Knowledge Distillation Bioinformatics Matryoshka Representation Learning

Open paper AskPDF arXiv

Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.

Abstract

Knowledge Distillation

Explore similar work

Reverse Distillation: Consistently Scaling Protein Language Model Representations

LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling