Cross-Granularity Representations for Biological Sequences: Insights from ESM and BiGCARP

Hanlin Xiao; Rainer Breitling; Eriko Takano; Mauricio A. Álvarez

cs.LGMar 21, 2026

Cross-Granularity Representations for Biological Sequences: Insights from ESM and BiGCARP

Authors: Hanlin Xiao, Rainer Breitling, Eriko Takano, Mauricio A. Álvarez

Organizations: Manchester Institute of Biotechnology, The University of Manchester, Manchester, United Kingdom · Department of Computer Science, The University of Manchester, Manchester, United Kingdom

Abstract

Recent advances in general-purpose foundation models have stimulated the development of large biological sequence models. While natural language shows symbolic granularity (characters, words, sentences), biological sequences exhibit hierarchical granularity whose levels (nucleotides, amino acids, protein domains, genes) further encode biologically functional information. In this paper, we investigate the integration of cross-granularity knowledge from models through a case study of BiGCARP, a Pfam domain-level model for biosynthetic gene clusters, and ESM, an amino acid-level protein language model. Using representation analysis tools and a set of probe tasks, we first explain why a straightforward cross-model embedding initialization fails to improve downstream performance in BiGCARP, and show that deeper-layer embeddings capture a more contextual and faithful representation of the model's learned knowledge. Furthermore, we demonstrate that representations at different granularities encode complementary biological knowledge, and that combining them yields measurable performance gains in intermediate-level prediction tasks. Our findings highlight cross-granularity integration as a promising strategy for improving both the performance and interpretability of biological foundation models.

Figures & tables

Fig. 1: Multi-granularity nature of biological sequences and its analogy to tokenization in NLP. a) Gene sequences at the nucleotide level can be transcribed and translated into amino acids, which can be further abstracted into Pfam domains by HMM scans. A biosynthetic gene cluster (BGC) can thus be represented as a sequence of Pfam domains, the granularity used in the BiGCARP model. b) Similarly, NLP sequences can be represented at different granularities, from characters through subwords (the “token” here) to words and sentences. Unlike statistical NLP tokenizers (e.g., byte-pair encoding), which are reversible, biological “tokenizers” are lossy but inject functional priors by providing higher-level abstractions.

Fig. 2: Overview of the BiGCARP model and ESM-based initialization. Left: A BGC is represented as a sequence of Pfam domains, which are embedded through an embedding matrix initialized either randomly or with ESM. BiGCARP processes these embeddings into last-layer representations, which are passed to a prediction head for masked language modeling and downstream probe tasks. Right: In ESM-based initialization, each Pfam domain is associated with a representative amino acid sequence. The sequence is encoded by the pre-trained ESM model, mean-pooled, and used to construct the embedding vector for that domain in BiGCARP. Blue blocks illustrate an example of domain PF1 receiving its embedding initialization $\{\text{PF}_{i}\}_{i=1}^{n}$ from its representative amino acid sequence (right), and progressing from its embedder-layer embeddings $\{\text{PF}_{iE}\}_{i=1}^{n}$ to its last-layer embeddings $\{\text{PF}_{iL}\}_{i=1}^{n}$ (left).

Fig. 3: Two-dimensional UMAP projection of BiGCARP last-layer embeddings. All Pfam domains in the training corpus are shown and colored by functional category. Functionally coherent clusters emerge, indicating that BiGCARP captures high-level functional information. Three representative clusters are selected and analyzed in the main text to illustrate their underlying themes.

Fig. 4: CKA self-similarity of the ESM-initialized model across layers. The four-period pattern reflects the model’s four-block architecture. A sharp change between layers 0 and 1 arises from the dilated convolution layer, while the low similarity between the first and last layers indicates limited retention of initial representations.

Fig. 5: Evolution of CKA similarity during training across layers and initialization schemes. The near-constant similarity of the ESM-initialized embedder to itself (blue) indicates that it is trainable but minimally reshaped during training. The red and purple curves show that, despite different initialization strategies, the similarity between initial and last layers converges to a similar level, suggesting that ESM initialization does not preserve additional information in the final layer. The low similarity of cross-model comparisons (green) provides a consistency check.

Embedding Strategy	Macro AUROC
ESM + BiGCARP (random-initialized, last-layer)	$\mathbf{0.917}^{+0.003}_{-0.003}$
ESM-initialized (last-layer)	$0.913^{+0.005}_{-0.005}$
Random-initialized (last-layer)	$0.912^{+0.005}_{-0.005}$
Pfam2Vec + Random Forest	$0.911^{+0.003}_{-0.003}$
ESM embeddings	$0.908^{+0.003}_{-0.003}$
Random-initialized (embedder)	$0.903^{+0.003}_{-0.003}$

TABLE I: BGC Product Class Prediction Performance

Property	Stachel (ESM)	BiGCARP	BiGCARP + Stachel
is_aromatic	$\mathbf{0.935}^{+0.007}_{-0.009}$	$0.813^{+0.008}_{-0.010}$	$0.872^{+0.005}_{-0.015}$
has_heterocycle	$\mathbf{0.955}^{+0.004}_{-0.008}$	$0.895^{+0.006}_{-0.006}$	$0.920^{+0.008}_{-0.010}$
high_polarity	$\mathbf{0.913}^{+0.004}_{-0.008}$	$0.620^{+0.011}_{-0.009}$	$0.843^{+0.011}_{-0.020}$
is_Val	$\mathbf{0.960}^{+0.003}_{-0.004}$	$0.909^{+0.004}_{-0.005}$	$0.937^{+0.008}_{-0.011}$
is_Gly	$\mathbf{0.980}^{+0.002}_{-0.004}$	$0.923^{+0.004}_{-0.005}$	$0.958^{+0.008}_{-0.011}$
is_canonical_aa	$\mathbf{0.946}^{+0.005}_{-0.006}$	$0.885^{+0.005}_{-0.009}$	$0.927^{+0.007}_{-0.008}$

TABLE II: A-Domain Substrate Property Prediction Performance (AUROC)

Embedding Strategy	AUROC
ESM + BiGCARP (mean-pooled)	$\mathbf{0.759}^{+0.165}_{-0.209}$
ESM embeddings	$0.659^{+0.216}_{-0.241}$
BiGCARP (mean-pooled)	$0.181^{+0.154}_{-0.130}$

TABLE III: Halogen Prediction Performance (AUROC)

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

When Do Biological Reasoning Models Use Their Biological Inputs?
Oct 1, 2026Ada Fang, Nikitha Thoduguli, Lukas Fesser +3Faithfulness of Language Model Explanations Causal Interventions in Language Models
Harvard University · Massachusetts Institute of Technology
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Jun 20, 2026Qizhi Pei, Zhimeng Zhou, Yi Duan +9Multimodal Understanding Multimodal Pretraining
1Gaoling School of Artificial Intelligence, Renmin University of China, 2OpenDataLab, Shanghai Artificial Intelligence Laboratory · 3Zhejiang University, 4Shanghai Innovation Institute · 2OpenDataLab, Shanghai Artificial Intelligence Laboratory, 3Zhejiang University, 4Shanghai Innovation Institute +5
Task- and dataset-specific information in protein language models
Aug 12, 2026Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2Protein Language Models Protein Representation Learning
Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Oct 1, 2026J/K move · Enter open · S save

When Do Biological Reasoning Models Use Their Biological Inputs?

Ada Fang, Nikitha Thoduguli, Lukas Fesser, Hanlin Zhang+2

Harvard University · Massachusetts Institute of Technology

Faithfulness of Language Model Explanations Causal Interventions in Language Models Bioinformatics Molecular Foundation Models

Open paper AskPDF arXiv

Biological reasoning models use post-training to connect LLMs to biological foundation model representations and biological text. Their benchmark accuracy is taken as evidence that LLMs reason over these inputs. We test this assumption in six biological reasoning models across DNA, protein, and single-cell tasks. We perturb one biological input while holding the query and other inputs fixed, construct evidence conflicts that pair the foundation model representation of one genome, protein, or cell with the text of another, fit linear probes to the representations the language model receives, and analyze reasoning traces against the biological inputs. Evo2 and ESM3 contribute little to BioReason and BioReason-Pro performance on the evaluated tasks. Shuffling the DNA sequence barely changes BioReason disease prediction accuracy, and in evidence conflicts the two models follow the text in 97.9% and 99.7% of cases. Linear probes trained on the Evo2 and ESM3 representations predict the task targets, so these foundation models encode information relevant to the task, but provide limited overall performance improvement to BioReason and BioReason-Pro. In contrast, foundation model inputs contribute to ChatNT, Prot2Text-V2, and CellWhisperer performance, and differentially expressed genes in the gene sentence contribute to Cell2Sentence-Scale performance. Across SFT and RL checkpoints of BioReason-Pro and 42 BioReason checkpoints, increases in accuracy do not imply greater performance contributions from biological inputs. BioReason traces misstate nucleotide changes, while BioReason-Pro traces describe functions omitted from final predictions under evidence conflicts. We find that current post-training strategies do not ensure that foundation model representations contribute to task performance.

Abstract

Representation Learning

Explore similar work

When Do Biological Reasoning Models Use Their Biological Inputs?

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language

Task- and dataset-specific information in protein language models