cs.LGFeb 26, 2026

Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models

Authors: Gal Pomerants, Yaniv Nikankin, Anja Reusch, Tomer Tsaban, Ora Schueler-Furman, Yonatan Belinkov

Organizations: Technion - Israel Institute of Technology · The Hebrew University of Jerusalem, Israel · Kempner Institute, Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Protein sequences are abundant in repeating segments, both as exact copies and as approximate segments with mutations. These repeats are important for protein structure and function, motivating decades of algorithmic work on repeat identification. Recent work has shown that protein language models (PLMs) identify repeats, by examining their behavior in masked-token prediction. To elucidate their internal mechanisms, we investigate how PLMs detect both exact and approximate repeats. We find that the mechanism for approximate repeats functionally subsumes that of exact repeats. We then characterize this mechanism, revealing two main stages: PLMs first build feature representations using both general positional attention heads and biologically specialized components, such as neurons that encode amino-acid similarity. Then, induction heads attend to aligned tokens across repeated segments, promoting the correct answer. Our results reveal how PLMs solve this biological task by combining language-based pattern matching with specialized biological knowledge, thereby establishing a basis for studying more complex evolutionary processes in PLMs.

Figures & tables

Figure 1: Visualization of the repeat identification mechanism. The model predicts the masked token by integrating repetition-related (left) and biological features (right). (I) First, relative-position attention heads attend to tokens located at fixed offsets ( ±n\pm n ) from the masked position, followed by the activation of biologically specialized neurons, such as neurons that selective for biochemically similar amino acids. (II) In middle layers, induction heads attend to the token aligned with the masked position in the other repeat instance and copy its information, enabling retrieval of the correct amino acid, while repetition neurons play an inhibitory role. (III) Finally, MLP neurons refine the final masked token distribution, with amino-acid-biased attention heads also contributing to the prediction.
ModelSyntheticIdenticalApproximate
ESM-399.8%99.0%79.0%
ESM-C99.9%99.4%79.7%
Table 1: Model accuracies on repeat prediction tasks. The analyzed PLMs show high performance in different settings of repeat identification, motivating our research.
Figure 2: ESM-3 circuit faithfulness scores. Across the three tasks, the discovered circuits achieve high faithfulness (above the 85% threshold) using a small fraction of model components.
Figure 3: Cross-Task Circuit Comparisons in ESM-3. We compare the IoU, recall and cross-task faithfulness of the circuits found for the three repeat tasks. The IoU (left) shows relatively high overlap between all three tasks. The recall (middle) measures the fraction of the ground-truth circuit (x-axis) recovered by the predicted circuit (y-axis), and shows that the synthetic and identical circuits are largely subsumed within each other and within the approximate-repeat circuit. The cross-task faithfulness (right) denotes the faithfulness of the circuit (y-axis) on data from another task variant (x-axis), with scores normalized per target task relative to the faithfulness of its own circuit. The results show that the approximate repeat circuit is the only one that functionally generalizes to the other settings.
Figure 4: Active attention patterns in ESM-3. (a–c) show example attention maps from three attention heads in the approximate-repeat circuit, for a single representative input: (a) fixed relative-position attention (diagonal); (b) induction attention between aligned repeat positions (two partial diagonals); and (c) amino-acid–biased attention (vertical). (d) Circuit attention heads are clustered and visualized using UMAP, colored by pattern type.
FeatureDescription
Induction scorePattern-matching and copying across aligned repeats.
Relative positionConsistent attention at fixed offsets.
Amino acid biasSelectivity for specific amino acids.
Repeat focus scoreSharper attention to repeat regions.
Attention sharpnessAttention distribution concentration.
Boundary attentionAttention to boundary tokens.
Table 2: Attention head features. Full details in Appendix I.2 .
Figure 5: Circuit faithfulness of subsets of neurons in ESM-3. 3% of MLP neurons per layer are sufficient to recover most circuit faithfulness (80/85%), indicating sparsity in the neuron basis.
Figure 6: (A) Examples of three task-relevant neurons in ESM-3: an amino-acid–specific neuron, a neuron that responds to substitutable amino acids according to BLOSUM62 groupings, and a repeat-focused neuron. We show token-level activations on an example sequence (left) and activation distributions over the full dataset (right). (B) AUROC scores for each neuron’s best-matching concept across ESM-3 layers. Each point is a neuron, plotted by layer and AUROC, and colored by concept category. “Biological (Other)” denotes groupings from IMGT physicochemical classes and secondary-structure propensities. (C) Number of neurons associated with “Biological (Other)” concepts in ESM-3 and ESM-C, restricted to neurons with AUROC ≥75%\geq 75\% .
Figure 7: Aggregated interactions between component groups in ESM-3. Larger scores indicate stronger influence. Induction heads and MLPs show the strongest influence on the logits, with weaker contributions from AA-biased heads. The input primarily affects non-induction components, and interactions are strongest between attention heads and MLPs, highlighting the central role of MLPs in mediating information flow.
Figure 8: Logit-lens analysis of ESM-3 component outputs. We measure the direct averaged contribution of each component group to the correct prediction at each layer. Induction heads and late MLPs are the major promoters of the correct answer, while other component groups only contribute indirectly.
Figure 9: Attribution analysis of neuron-group importance across layers in ESM-3. Biochemical-similarity neurons are most important in early layers, amino-acid neurons contribute stronger in later layers, and repeat-related neurons often show inhibitory effects, reflected by negative contributions across layers.
Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

TaskMasked InputRepeat AlignmentEligible mask positions
Synthetic...NFDIQREHYKTEWYN... ...NFDI_REHYKTEWYN...NFDIQREHYKTEWHRYN ||||||||||||||||| NFDIQREHYKTEWHRYNAll positions
Identical...MKTAYIAKQRQISFVHFS... ...MKTA_IAKQRQISFVHFS...MKTAYIAKQRQISFVKSHFS |||||||||||||||||||| MKTAYIAKQRQISFVKSHFSAll positions
Approximate (Substitution-Adjacent)...PYTSAVTQTR... ...PYRA_VTQTR...PYTSAVTQTR ||..|||||| PYRAAVTQTRIdentical positions adjacent to substitutions (Y, A in example)
Approximate (Indel-Adjacent)...GNAASATKLQTSR... ...GNAASAT_ATQDSR...GNAASATK-LQTSR |||||||| | .|.|| GNAASATKATQDSRIdentical positions adjacent to gaps (K in example)
Appendix
Table 3: Examples of masked inputs for each task. Masked Input shows the masked residue as an underscore; ellipses ( ... ) denote non-repeat regions not shown for brevity. Repeat Alignment marks exact matches ( | ), substitutions ( . ), and gaps ( - ). Eligible Positions specifies which repeat positions may be masked under each task definition.
ModelSubst. rateRepeat length
  1–1011–1516–2021–2526+
ESM-300 – 25%25\%0.770.770.940.940.960.960.980.980.980.98
2525 – 50%50\%0.450.450.630.630.850.850.920.920.960.96
ESM-C00 – 25%25\%0.750.750.940.940.960.960.960.960.970.97
2525 – 50%50\%0.510.510.650.650.850.850.920.920.940.94
Appendix
Table 4: Approximate-repeat accuracy by repeat length and substitution rate. Accuracy is averaged within each bin.
Figure 10: Effect of substitution rate on induction-head contributions to repeat completion. In both models, induction-head attribution remains stable or initially increases up to a length-dependent substitution rate, after which it decreases sharply. This transition occurs earlier for shorter repeats, suggesting that the induction-based repeat-completion mechanism is less robust when the repeat signal is short or highly divergent.
Figure 11: ESM-C circuit faithfulness scores. Across all three task settings, the discovered circuits achieve high faithfulness using approximately 25%25\% of model components.
Figure 12: Cross-task circuit comparisons in ESM-C. All circuits show similar structural overlap, while only the approximate repeats circuit exhibits strong functional generalization to the other tasks.
Figure 13: Attention patterns in ESM-C. Example attention maps from three attention heads in ESM-C approximate repeats circuit. for a single representative input, illustrating the same recurring patterns we observe in ESM-3: Left: fixed relative position attention, producing a diagonal pattern; Middle: induction attention between aligned repeat positions, resulting in two partial diagonals due to bidirectional alignment; Right: amino acid biased attention, producing a vertical pattern.
Figure 14: UMAP projection of clustered attention heads in ESM-C, colored by pattern type.
Figure 15: Circuit faithfulness with a subset of neurons in ESM-C. Retaining approximately 7% of MLP neurons per layer achieves 80%80\% faithfulness.
Figure 16: AUROC scores for each neuron’s best-matching concept across ESM-C layers. Each point corresponds to a neuron, plotted by layer and AUROC score, and colored by concept category. “Biological (Other)” denotes groupings from IMGT physicochemical classes and secondary-structure propensities.
Figure 17: Aggregated interactions between component groups in ESM-C. Positive interactions between source and destination component groups, reflecting connections that support task performance.
Figure 18: Logit-lens analysis of ESM-C component outputs. Induction heads promote the correct token in the middle layers, while final-layer MLPs continue to increase the prediction confidence.
Figure 19: Attribution-based analysis of neuron-group importance across layers in ESM-C. Biochemical similarity and amino-acid neurons are most important in mid layers, with amino-acid neurons contributing more strongly in the final layer. Repeat-related neurons exhibit inhibitory effects, indicated by negative contributions across multiple layers.
Counterfactual TypeInput Example
Original Input...RLRHQLSFYG _ PFALG...KLRHQLSFYGVAFALG...
100% BLOSUM (Chosen Strategy)...RLRHQLSFYG _ PFALG... RIKYEIAYFAISYSIA ...
Permutation...RLRHQLSFYG _ PFALG... QLFHYGARFLKLSAVG ...
100% BLOSUM Opposite...RLRHQLSFYG _ PFALG... CDCCCDWPDIRWPWDI ...
100% Mask...RLRHQLSFYG _ PFALG... ______________ ...
50% BLOSUM...RLRHQLSFYG _ PFALG...KLRHQ IAYF G ISY AL A ...
Appendix
Table 5: Examples of counterfactual strategies applied to approximate repeats. Only the repeated segments are shown for brevity. The masked position and corrupted amino acids are highlighted in blue. Underscores indicate masked tokens.
Figure 20: Effect of counterfactual corruption strategies on ESM-3 prediction performance across repeat tasks.
Figure 21: Component-level circuit evaluation for different counterfactual strategies on ESM-3. Faithfulness is shown as a function of the fraction of included components. Results are averaged across random seeds.
Figure 22: Cross-counterfactual recall of discovered circuits. Recall measures the fraction of components in the target circuit that are recovered by the source circuit. Circuit sizes (number of components) achieving 85%85\% faithfulness are shown in parentheses, averaged across seeds.
Figure 23: Cross-counterfactual faithfulness of discovered circuits. Rows correspond to counterfactuals used for circuit discovery, columns correspond to counterfactuals used in evaluation. Faithfulness is normalized per column, relative to the circuit discovered under that counterfactual. Circuit sizes (number of components) achieving 85%85\% faithfulness are shown in parentheses.
Table 6: Example ESM-3 attention patterns on a protein with two repeats (UniProt A0A2M8A3Y9): MVKIWGREDG (1–10) and MVKIWGKKDG (63–72). Repeated regions are marked by red dashed vertical and horizontal lines. Panels (e,f) show AA-biased heads that activate primarily within repeats, motivating the repeat focus score.
Table 7: Example attention patterns from ESM-C on a UniRef50 protein (A0A8X6HTE9_TRICU). The sequence contains two approximate repeats, TRKGELSRRGCSSG (positions 4–17) and TRKGMLSRRECSSG (positions 60–73). Repeated regions are indicated by red dashed vertical and horizontal lines, marking the start and end of the repeat span along the key (x-axis) and query (y-axis), respectively. (a,b) Heads attending to fixed relative positional offsets. (c,d) Induction-like heads attending between aligned positions across repeated segments. (e,f) Amino-acid–biased heads focusing on specific amino acids within repeated regions, as in ESM-3.
FeatureFormal DefinitionDescription
Induction ScoreInductionh=12(Copyingh+PMh)Copyingh=1∣D∣∑s∈D(P~h,m(s)(s) ⁣(tj∗(s))∑t∈U(s)P~h,m(s)(s)(t))where j∗(s)=arg⁡max⁡j∈{0,…,L+1}Ah(s)[m(s),j]PMh=1∣D∣∑s∈D(1∣Rpairs(s)∣∑(q,k)∈Rpairs(s)Ah(s)[q,k])\text{Induction}_{h}=\tfrac{1}{2}\left(\text{Copying}_{h}+\text{PM}_{h}\right)\newline \newline \text{Copying}_{h}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\left(\frac{\widetilde{P}_{h,m(s)}^{\text{(s)}}\!\left(t_{j^{\ast}(s)}\right)}{\sum_{t\in\mathcal{U}(s)}\widetilde{P}_{h,m(s)}^{\text{(s)}}(t)}\right)\newline \text{where }j^{\ast}(s)=\arg\max_{j\in\{0,\dots,L+1\}}A_{h}^{(s)}[m(s),j]\newline \newline \text{PM}_{h}=\newline \frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\left(\frac{1}{|\mathcal{R}_{\text{pairs}}(s)|}\sum_{(q,k)\in\mathcal{R}_{\text{pairs}}(s)}A_{h}^{(s)}[q,k]\right)Measures induction-like behavior following prior work on decoder-only models ( Olsson et al., 2022 ; Ren et al., 2024 ; Bansal et al., 2023 ) , which defines induction behavior as the combination of pattern-matching and copying. The Copying score quantifies whether the head promotes the token it attends to most strongly at the masked position, while the Pattern Matching component measures attention between aligned tokens across the two repeat instances. Both scores are first computed at the per-sequence level and averaged across all sequences. The Induction Score is then defined as the mean of these two aggregated quantities, yielding a single score per head. This score is motivated by manual inspection of attention patterns that closely resemble induction behavior previously identified in large language models.
Relative Position ScoreXq(s)  =  (arg⁡max⁡kAh(s)[q,k])−qLμh(s)  =  1L∑q=1LXq(s)RPh(s)  =  1L∑q=1L(Xq(s)−μh(s))2RPh  =  1−1∣D∣∑s∈DRPh(s)X_{q}^{(s)}\;=\;\frac{(\arg\max_{k}A_{h}^{(s)}[q,k])-q}{L}\newline \mu_{h}^{(s)}\;=\;\tfrac{1}{L}\sum_{q=1}^{L}X_{q}^{(s)}\newline \text{RP}_{h}^{(s)}\;=\;\sqrt{\tfrac{1}{L}\sum_{q=1}^{L}\left(X_{q}^{(s)}-\mu_{h}^{(s)}\right)^{2}}\newline \newline \text{RP}_{h}\;=\;1-\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\text{RP}_{h}^{(s)}For each sequence ss and query position qq , Xq(s)X_{q}^{(s)} denotes the normalized relative offset between the query and the key position receiving maximal attention from head hh . The per-sequence score RPh(s)\text{RP}_{h}^{(s)} is computed as the standard deviation of Xq(s)X_{q}^{(s)} across query tokens, with μh(s)\mu_{h}^{(s)} denoting the mean offset. Offsets are normalized by the sequence length LL to allow comparison across sequences of different lengths, and BOS/EOS positions are excluded as queries. The final score RPh\text{RP}_{h} is obtained by averaging RPh(s)\text{RP}_{h}^{(s)} over the dataset D\mathcal{D} and inverting it ( 1−⋅1-\cdot ), so that higher values indicate heads that attend to a more consistent relative position. This metric is motivated by qualitative observations of heads attending to fixed relative offsets (e.g., ±n\pm n tokens).
Attn. Amino-Acid BiasAA_Biash,a(s)=1L∑q=1L∑k∈Ka(s)Ah(s)[q,k]AA_Biash,a=1∣D∣∑s∈DAA_Biash,a(s)Ph(a)=AA_Biash,a∑a′∈AAAA_Biash,a′AA_Scoreh=JSD⁡ ⁣(Ph  ∥  Pdata)\text{AA\_Bias}_{h,a}^{(s)}=\frac{1}{L}\sum_{q=1}^{L}\sum_{k\in K_{a}(s)}A_{h}^{(s)}[q,k]\newline \text{AA\_Bias}_{h,a}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\text{AA\_Bias}_{h,a}^{(s)}\newline P_{h}(a)=\frac{\text{AA\_Bias}_{h,a}}{\sum_{a^{\prime}\in\mathcal{AA}}\text{AA\_Bias}_{h,a^{\prime}}}\newline \text{AA\_Score}_{h}=\operatorname{JSD}\!\left(P_{h}\;\middle\|\;P_{\text{data}}\right)For each amino acid aa , we compute the average attention mass that head hh assigns from sequence-token queries to all key positions containing aa , and average this quantity across the dataset, yielding one feature per amino acid. These features are normalized to form an amino-acid attention distribution for each head, which is compared to the dataset amino-acid frequency distribution using Jensen–Shannon divergence. This score is motivated by qualitative observations of attention patterns, including global amino-acid biases, where a head broadly attends to all occurrences of a subset of amino acids, as well as more localized, motif-like patterns, where attention is concentrated on specific amino acids at particular positions.
Repeat FocusRFh(s)=log⁡1∣R(s)∣∑q∈R(s)max⁡kAh(s)[q,k]1∣N(s)∣∑q∈N(s)max⁡kAh(s)[q,k]RFh=1∣D∣∑s∈DRFh(s)\text{RF}_{h}^{(s)}=\log\frac{\frac{1}{|\mathcal{R}(s)|}\sum_{q\in\mathcal{R}(s)}\max_{k}A_{h}^{(s)}[q,k]}{\frac{1}{|\mathcal{N}(s)|}\sum_{q\in\mathcal{N}(s)}\max_{k}A_{h}^{(s)}[q,k]}\newline \text{RF}_{h}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\text{RF}_{h}^{(s)}Measures how differently head hh behaves when the query comes from a repeat segment compared to non-repeat regions, by partitioning query positions into repeat queries R(s)\mathcal{R}(s) and non-repeat queries N(s)\mathcal{N}(s) . For each query, the maximum attention peak across keys is computed and averaged separately over the two sets. The logarithm of their ratio centers the measure at zero: positive values indicate sharper focus on repeat queries, negative values indicate sharper focus on non-repeat queries, and values near zero indicate similar behavior across regions. The score is computed per sequence and averaged across all sequences in the dataset. This score is motivated by manual observations that, beyond induction heads, additional heads tend to exhibit stronger activation for repeat queries.
Attention to BOS/EOSAhBOS=1∣D∣∑s∈D1L∑q=1LAh(s)[q, 0]AhEOS=1∣D∣∑s∈D1L∑q=1LAh(s)[q, L+1]AhBOS/EOS=12(AhBOS+AhEOS)\text{A}^{\text{BOS}}_{h}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\frac{1}{L}\sum_{q=1}^{L}A_{h}^{(s)}[q,\,0]\newline \text{A}^{\text{EOS}}_{h}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\frac{1}{L}\sum_{q=1}^{L}A_{h}^{(s)}[q,\,L+1]\newline \newline \text{A}^{\text{BOS/EOS}}_{h}=\tfrac{1}{2}\left(\text{A}^{\text{BOS}}_{h}+\text{A}^{\text{EOS}}_{h}\right)Measures the average attention weight that head hh assigns from sequence-token query positions to the sequence-boundary positions corresponding to the beginning of the sequence (BOS) and the end of the sequence (EOS). Throughout, positions 00 and L+1L+1 denote the BOS and EOS boundary positions, respectively. Both scores are computed at the per-example level and averaged across all sequences. This metric is motivated by manual inspection of attention patterns, where we observed several heads that attend to boundary positions in a subset of proteins examined qualitatively. Prior work suggests that attention to sequence-boundary positions can be associated with “no-op” behavior or approximate if–else mechanisms, in which heads default to boundary positions when no salient pattern is detected ( Clark et al., 2019 ; Vig and Belinkov, 2019 ; Barbero et al., 2025 ) .
Attention EntropyENTh(s)=1L∑q=1L−∑k=0L+1Ah(s)[q,k]log⁡Ah(s)[q,k]log⁡(L+2)ENTh=1∣D∣∑s∈DENTh(s)\text{ENT}_{h}^{(s)}=\frac{1}{L}\sum_{q=1}^{L}\frac{-\sum_{k=0}^{L+1}A_{h}^{(s)}[q,k]\log A_{h}^{(s)}[q,k]}{\log(L+2)}\newline \text{ENT}_{h}=\frac{1}{|\mathcal{D}|}\sum_{s\in\mathcal{D}}\text{ENT}_{h}^{(s)}Computes the average normalized Shannon entropy of the attention distribution of head hh over sequence-token queries, excluding BOS/EOS as queries but including them as keys. For each query, entropy is normalized by log⁡(L+2)\log(L+2) , corresponding to the tokenized sequence length, yielding values in [0,1][0,1] and enabling comparison across sequences of different lengths. The score is averaged across all sequences in the dataset. High entropy indicates that the head spreads attention broadly across many tokens, while low entropy indicates that the head focuses attention sharply on a few tokens. This metric is motivated by empirical observations that relative-position and induction heads typically exhibit lower entropy than other heads, making attention entropy useful for separating distinct head behaviors.
Appendix
Table 8: Summary of attention head features computed per protein sequence
Figure 24: Cluster selection metrics for attention-head clustering. We report inertia (Elbow method, left) and Silhouette scores (right) for k∈[2,10]k\in[2,10] for ESM-3 (top) and ESM-C (bottom).
ClusterHeads
C0C_{0}a0.h7, a0.h17, a3.h9, a3.h12, a3.h14, a5.h1, a5.h14, a5.h21, a7.h19, a7.h21, a8.h9, a9.h15, a9.h17, a15.h10, a22.h2, a22.h9, a22.h10, a22.h13, a24.h9, a25.h13, a25.h14, a25.h16, a26.h0, a26.h2, a27.h7, a27.h11, a27.h12, a27.h13, a28.h10, a28.h11, a30.h2, a30.h22, a30.h23, a31.h9, a32.h4, a32.h14, a32.h21, a34.h23, a35.h1, a35.h3, a35.h20, a36.h2, a36.h3, a36.h4, a36.h9, a36.h10, a36.h15, a36.h20, a37.h0, a37.h2, a37.h4, a37.h9, a37.h19, a39.h7, a39.h8, a39.h13, a39.h15, a39.h17, a39.h22, a40.h1, a40.h8, a40.h13, a40.h19, a40.h21, a41.h2, a41.h3, a41.h7, a41.h8, a41.h14, a41.h20, a43.h2, a43.h3, a43.h4, a44.h1, a44.h2, a44.h9, a44.h11, a44.h16, a44.h18, a45.h0, a45.h3, a45.h4, a45.h17, a45.h23, a46.h1, a46.h2, a46.h3, a46.h4, a46.h8, a46.h16, a47.h5, a47.h8, a47.h10, a47.h12, a47.h14, a47.h15, a47.h17, a47.h21, a47.h22
C1C_{1}a23.h5, a23.h23, a28.h13, a29.h1, a38.h10, a38.h22, a42.h5
C2C_{2}a0.h4, a24.h16, a24.h19, a25.h23, a26.h19, a27.h5, a31.h8, a31.h17, a33.h23, a34.h9, a35.h10, a35.h21, a37.h8, a37.h10, a37.h15, a38.h0, a38.h18, a39.h3, a39.h6, a39.h10, a40.h7, a40.h11, a40.h16, a40.h17, a40.h22, a41.h13, a41.h17, a42.h10, a45.h2, a45.h6, a45.h9, a45.h10, a45.h16, a46.h10, a46.h17, a46.h22, a47.h2, a47.h3, a47.h6, a47.h7, a47.h18, a47.h19, a47.h20
Appendix
Table 9: Cluster assignments for k=3k=3 obtained from kk -means clustering over attention-head features for ESM-3.
ClusterHeads
C0C_{0}a0.h7, a0.h15, a0.h16, a0.h17, a1.h3, a1.h10, a7.h1, a7.h7, a8.h6, a8.h16, a9.h3, a9.h9, a10.h7, a10.h8, a11.h8, a14.h5, a15.h0, a15.h2, a15.h9, a15.h10, a15.h15, a15.h17, a16.h2, a16.h5, a16.h12, a16.h13, a17.h0, a18.h2, a18.h6, a18.h8, a18.h9, a18.h13, a18.h14, a18.h17, a19.h7, a20.h1, a20.h6, a20.h13, a21.h6, a22.h8, a22.h10, a22.h12, a23.h13, a23.h15, a24.h0, a24.h1, a24.h2, a24.h3, a24.h10, a25.h16, a26.h1, a27.h1, a27.h16, a29.h12, a29.h15, a34.h0, a34.h9, a35.h0, a35.h3, a35.h10, a35.h11, a35.h12
C1C_{1}a12.h2, a13.h5, a17.h4, a17.h12, a19.h8, a19.h14, a19.h16, a21.h0, a28.h3, a29.h13, a30.h12, a35.h16
C2C_{2}a4.h14, a13.h12, a14.h17, a16.h3, a16.h11, a17.h2, a17.h8, a18.h16, a20.h7, a21.h14, a25.h12, a26.h2, a26.h16, a27.h0, a28.h17, a29.h5, a29.h10, a30.h15, a33.h10, a34.h8, a34.h11, a34.h15, a35.h1, a35.h2, a35.h5, a35.h13, a35.h15, a35.h17
Appendix
Table 10: Cluster assignments for k=3k=3 obtained from kk -means clustering over attention-head features for ESM-C.
ClusterSizeInductionRel. Pos.AA BiasAttn. BOS/EOSEntropyResidual Contr.Vocab. EntropyContext Sens.Repeat Focus
C0C_{0}990.0260.0260.9920.9920.0430.0430.0060.0060.3240.3240.2880.2880.4780.4780.0800.080−0.002-0.002
C1C_{1}70.6500.6500.6670.6670.0410.0410.0510.0510.5370.5370.4620.4620.150.150.7610.7611.0031.003
C2C_{2}430.0450.0450.8170.8170.1280.1280.0430.0430.7650.7650.1930.1930.5640.5640.2830.2830.1080.108
Appendix
Table 11: Feature profiles of attention-head clusters in ESM-3. Values are reported before standardization.
ClusterSizeInductionRel. Pos.AA BiasAttn. BOS/EOSEntropyResidual Contr.Vocab. EntropyContext Sens.Repeat Focus
C0C_{0}620.0260.0260.9870.9870.0350.0350.0080.0080.3060.3060.2970.2970.6020.6020.0860.0860.0510.051
C1C_{1}120.4930.4930.6960.6960.0400.0400.0720.0720.5030.5030.3220.3220.4450.4450.7590.7590.8570.857
C2C_{2}280.0410.0410.7510.7510.0830.0830.0760.0760.7060.7060.2150.2150.6500.6500.2920.2920.0840.084
Appendix
Table 12: Feature profiles of attention-head clusters in ESM-C. Values are reported before standardization.
ModelHeadRepeat FocusCluster
ESM-3a26.h190.701C2C_{2}
ESM-3a24.h160.628C2C_{2}
ESM-3a27.h50.618C2C_{2}
ESM-3a46.h220.565C2C_{2}
ESM-3a31.h80.562C2C_{2}
ESM-3a38.h00.541C2C_{2}
Appendix
Table 13: Non-induction attention heads (Cluster C2C_{2} ) with elevated repeat focus. Shown are attention heads from Cluster C2C_{2} with outlier values of the Repeat Focus feature, identified using an IQR-based threshold ( score>Q3+1.5⋅IQR\text{score}>Q_{3}+1.5\cdot\mathrm{IQR} ), for both ESM-3 and ESM-C. Note that, based on the score definition, values above zero indicate that query tokens in repeated regions tend to assign stronger attention to specific key positions than non-repeated region queries, with higher magnitude indicating a stronger repeat-focus phenomenon.
Concept / Concept GroupDescription
Repeat TokensWe define a repeat-based concept. A token belongs to this concept if it appears within a repeated segment in one of the repeats present in the dataset.
Aligned Repeat Token to Mask PositionWe define an alignment-based concept. A token belongs to this concept if it is aligned to the masked position, i.e., the token from which the model is expected to retrieve information to predict the masked token. This concept is motivated by manual inspection, which revealed consistent neuron patterns targeting this aligned position across multiple proteins.
Amino Acid IdentityWe define amino-acid–based concepts by assigning a distinct concept to each of the 20 standard amino acids. A token is assigned to a concept if it represents that specific amino acid. This corresponds to a total of 20 concepts.
BLOSUM62-Based ConceptsWe construct groups of substitutable amino acids using the BLOSUM62 substitution matrix ( Henikoff and Henikoff, 1992 ) . We view the matrix as a weighted graph over amino acids and identify cliques in which all pairwise substitution scores are greater than or equal to zero, corresponding to substitutions that are neutral or favorable. Each such clique is treated as a distinct concept. This procedure yields a total of 110 BLOSUM-based concepts. Although many of these cliques overlap, this does not pose an issue for our analysis, since each neuron is assigned to the best-matching concept according to its AUROC score.
Polarity (IMGT)We define two polarity-based concepts according to the IMGT classification: polar and non-polar . A token is assigned to one of these concepts based on the polarity of the amino acid it represents.
Hydropathy (IMGT)We define three hydropathy-based concepts according to the IMGT classification: hydrophobic , neutral , and hydrophilic . A token is assigned to one of these concepts based on the hydropathy class of the amino acid it represents.
Appendix
Table 14: Concept definitions used for neuron analysis.
Figure 25: Example sequence visualizations for a neuron in ESM-3 (layer 38, neuron 2850) selective for the aligned repeat token to the masked position. The neuron activates on the token aligned with the mask (denoted by “?”), corresponding to the correct retrieval target. Repeat tokens are highlighted in green. UniProt accessions: A0A0P7XBW1 (top), B0DTM3 (bottom).
Figure 26: Example sequence visualization for a repeat-selective neuron in ESM-C (layer 24, neuron 2224). The neuron exhibits positive activations (red) at positions corresponding to approximate repeat tokens, highlighted in green. UniProt accession: A0A8X6HTE9.
Figure 27: Example sequence visualization for a helix-breaker–selective neuron in ESM-3 (layer 2, neuron 3282). The neuron exhibits strong positive activations (red) on proline (P) and glycine (G), amino acids known to disrupt α\alpha -helical secondary structure. UniProt accession: B0DTM3.
Figure 28: Example sequence visualization for an aromatic-ring–selective neuron in ESM-3 (layer 35, neuron 3011). The neuron exhibits strong positive activations (red) on aromatic amino acids (F, Y, H, W). UniProt accession: A0A0P7XBW1.
Figure 29: Example sequence visualization for a hydrogen-donor–selective neuron in ESM-C (layer 7, neuron 907). The neuron exhibits strong positive activations (red) on amino acids R, K, and W, consistent with hydrogen-donor side-chain chemistry. UniProt accession: A0A2M8A3Y9.
Figure 30: Example visualizations of bidirectional amino-acid neurons in ESM-3 (top) and ESM-C (bottom), showing strong but oppositely signed responses to pairs of amino acids.
Figure 31: Example sequence visualizations for neurons selective for the BLOSUM62 substitution clique {D, E, N} in ESM-3 (top) and ESM-C (bottom).
Figure 32: Example sequence visualizations for neurons selective for special tokens in ESM-3 (top) and ESM-C (bottom). Both neurons exhibit strong, selective activations to the BOS token, with weaker activations to the EOS token. In the visualizations, < denotes the BOS token and > denotes the EOS token.
More in this topic

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Explore similar work

CardsList
Feb 24, 2026

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Anna Hart, Chi Han, Jeonghwan Kim +2

Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Protein Language ModelsProtein Representation LearningBioinformaticsMolecular Property Prediction
AskPDF
Aug 12, 2026

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Protein Language ModelsProtein Representation LearningLanguage Model Probing
AskPDF
Oct 19, 2026

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Saurav Jha, Qianli Liao, Robert Vernon +2

Mila – Quebec AI Institute · Polytechnique Montréal · Center for Brains, Minds, and Machines +4

Protein Language ModelsRecursive Language ModelsAdaptive ComputationProtein Structure PredictionMasked Language Modeling
AskPDF