Conditioning Protein Generation via Hopfield Pattern Multiplicity

Jeffrey D. Varner

cs.LGMar 20, 2026

Conditioning Protein Generation via Hopfield Pattern Multiplicity

Authors: Jeffrey D. Varner

Organizations: Robert Frederick Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850

Abstract

Small protein-family alignments often contain a subset of interest but not enough labeled data to train a conditional generator. We condition a training-free stochastic-attention sampler by adding one multiplicity ratio to its logits. Increasing this ratio shifts generation from the full family toward the designated subset. For unit-norm memories, the resulting Boltzmann distribution is exactly a Gaussian mixture whose component weights are set by the multiplicities. This result separates exact conditioning in latent space from losses caused by sampling, PCA reconstruction, and sequence decoding. Across five Pfam families, attention followed the analytic target, but recovery of single-residue markers depended on how well PCA separated the designated and background sequences. A matched weighted profile HMM reproduced these markers more directly, while stochastic attention gave lower ESM2 pseudo-perplexity in the Kunitz comparison. Using a curated set of 23 omega-conotoxin sequences as the target subset produced diverse sequences that preserved the cysteine scaffold and Tyr13 and shifted other residues toward the designated set. These sequences are candidates for experimental testing; they do not establish binding.

Figures & tables

Family	Source	$K$	$L$	$d$	$K_{\mathrm{des}}$	Readout	$f_{\mathrm{des}}$	$S$	$f_{\mathrm{obs}}^{\mathrm{hard}}$	$\Delta$
WW	PF00397	420	31	186	69	Spec. loop	0.16	0.11	$1.000\pm 0.000$	$0.660\pm 0.016$
Forkhead	PF00250	246	87	172	122	H/N at H3	0.50	0.17	$1.000\pm 0.000$	$0.267\pm 0.028$
Kunitz	PF00014	99	53	80	32	P1 K/R	0.32	0.20	$1.000\pm 0.000$	$0.409\pm 0.029$
SH3	PF00018	55	48	46	33	Trp	0.60	0.34	$1.000\pm 0.000$	$0.009\pm 0.003$
Homeobox	PF00046	136	57	101	102	Gln at pos. 50	0.75	0.42	$1.000\pm 0.000$	$0.054\pm 0.011$
$\omega$ -Conotoxin	SwissProt	74	26	34	23	Tyr13	0.31	0.78	$0.979\pm 0.006$	$0.158\pm 0.035$

Table 1: Cross-family summary of all protein families tested. For each family: source database, number of sequences (

K

), aligned length (

L

), PCA dimensionality (

d

, capturing 95% variance), designated subset size (

K_{\mathrm{des}}

), sequence readout, designated input fraction (

f_{\mathrm{des}}

), separation index (

S

), and observed readout fraction under hard curation (

f_{\mathrm{obs}}^{\mathrm{hard}}

). For the five Pfam families,

\Delta

is the calibration gap at

\rho=500

. The conotoxin designation is accession based, whereas its readout is Tyr13, so its

\Delta

compares different quantities. All six families were tested with both multiplicity weighting and hard curation.

Condition	$\beta$	P1 K/R $\uparrow$	Novelty
Unconditional ( $b=0$ )	3.8	$0.39\pm 0.02$	0.42
Multiplicity ( $f_{\mathrm{eff}}=0.99$ )	7.7	$0.56\pm 0.02$	0.40
Hard mask ( $b=-\infty$ ), at $\beta^{*}$	3.2	$0.50\pm 0.02$	0.43
Hard mask ( $b=-\infty$ ), $\beta=512$	512	$1.00\pm 0.00$	0.16
Hard curation (subset basis)	3.2	$1.00\pm 0.00$	0.32

Table 2: Kunitz P1 K/R recovery across the conditioning axis. Designated subset is K/R at the P1 position (

K_{\mathrm{des}}=32

). All conditions use the full-family PCA basis except hard curation, which rebuilds the basis from the designated subset. The hard mask (

b=-\infty

on background) is the

\rho\to\infty

endpoint of the multiplicity axis at fixed

\beta

. Novelty is one minus the mean nearest-neighbor identity to the training alignment. P1 K/R is the chain-level mean

\pm

standard error over 30 chains; novelty standard errors are below 0.01.

Family	( $K$ , $K_{\mathrm{des}}$ , $S$ )	$\rho$	$f_{\mathrm{eff}}$	$\bar{a}_{\mathrm{des}}$	$f_{\mathrm{obs}}$	$\Delta$	$D$
WW	(420, 69, 0.11)	1	0.164	$0.170\pm 0.006$	$0.144\pm 0.016$	0.02	0.686
		10	0.663	$0.659\pm 0.013$	$0.223\pm 0.028$	0.44	0.670
		100	0.952	$0.952\pm 0.003$	$0.291\pm 0.019$	0.66	0.652
		500	0.990	$0.990\pm 0.001$	$0.330\pm 0.016$	0.66	0.631
Forkhead	(246, 122, 0.17)	1	0.496	$0.496\pm 0.006$	$0.565\pm 0.013$	-0.07	0.485
		10	0.908	$0.907\pm 0.005$	$0.612\pm 0.019$	0.30	0.468

Table 3: Per-family

\rho

sweep: mean attention weights track

f_{\mathrm{eff}}

closely; decoded marker fidelity varies across PCA geometries. Selected

\rho

values for each family.

f_{\mathrm{eff}}

: effective designated fraction from multiplicity weights;

\bar{a}_{\mathrm{des}}

: mean softmax attention on designated patterns;

f_{\mathrm{obs}}

: mean hard-decoded marker fraction across five independent replicates (620 retained samples per replicate); error bars are replicate s.d.;

D

: mean pairwise diversity. In all families

\bar{a}_{\mathrm{des}}\approx f_{\mathrm{eff}}

within finite-sample uncertainty, but

f_{\mathrm{obs}}

converges fastest for Homeobox (

S=0.42

) and slowest for WW (

S=0.11

), reflecting the observed latent-to-sequence calibration gap. In conotoxin, designation is accession based while

f_{\mathrm{obs}}

measures Tyr13, so the difference is descriptive rather than a calibration gap.

Family	$f_{\mathrm{eff}}$	SA marker	HMM marker	SA $D$	HMM $D$
Kunitz	0.996	$0.587\pm 0.029$	$0.996\pm 0.002$	0.510	0.590
SH3	0.999	$0.989\pm 0.003$	$0.999\pm 0.001$	0.515	0.655
WW	0.990	$0.330\pm 0.016$	$0.988\pm 0.004$	0.631	0.628
Homeobox	0.999	$0.945\pm 0.011$	$1.000\pm 0.001$	0.536	0.605
Forkhead	0.998	$0.731\pm 0.028$	$0.997\pm 0.001$	0.402	0.562
$\omega$ -Conotoxin	0.996	$0.838\pm 0.035$	$0.834\pm 0.013$	0.443	0.381

Table 4: Matched multiplicity-weighted profile-HMM benchmark at

\rho=500

. Both methods receive the same cleaned alignment, designation labels, and relative sequence weights. Marker fractions are mean

\pm

s.d. across five independent replicates of 620 sequences.

D

is mean pairwise sequence diversity. The profile HMM transfers the selected single-position marginal directly in the five Pfam splits; SA additionally passes through its PCA decoder. For conotoxin,

f_{\mathrm{eff}}

refers to accession designation and the marker column reports Tyr13.

Source	pLDDT	TM	PPL	P1 K/R	KL ( $\times 10^{3}$ )	$D$
Stored (natural)	$89.3\pm 3.2$	$0.83\pm 0.02$	$4.97\pm 0.98$	0.32	0	0.63
SA (full family)	$90.4\pm 1.2$	$0.84\pm 0.01$	$4.78\pm 0.64$	$0.38\pm 0.03$	$7.6\pm 0.9$	0.56
SA (K/R-positive)	$90.7\pm 1.1$	$0.83\pm 0.01$	$4.72\pm 0.60$	$1.00\pm 0.00$	$18.7\pm 0.9$	0.50
SA (K/R-negative)	$91.0\pm 1.2$	$0.85\pm 0.01$	$4.73\pm 0.72$	$0.00\pm 0.00$	$11.2\pm 0.2$	0.53
HMM emit (HMMER3)	$60.4\pm 13.1$	$0.60\pm 0.19$	$16.1\pm 4.2$	$0.17\pm 0.04$	$27.1\pm 2.1$	0.90
HMM weighted ( $\rho=500$ )	—	—	$6.54\pm 0.89$	$0.996\pm 0.002$	$5.6\pm 0.4$	0.590

Table 5: Structure-model, language-model, and sequence-marker evaluation on the Kunitz domain. pLDDT: ESMFold predicted confidence (0–100;

>70

= confident). TM: TM-score against 1BPI chain A (

>0.5

is a conventional same-fold reference). PPL: ESM2-650M masked-marginal pseudo-perplexity (lower = more plausible). P1 K/R: fraction with Lys/Arg at the primary trypsin contact. KL: amino acid composition divergence (

\times 10^{3}

D

: mean pairwise diversity. Structural and LM values are mean

\pm

s.d. across 50 sequences; sequence-level metrics are mean

\pm

s.d. across 5 replicates. Structure-model scores were not computed for the additional weighted profile-HMM benchmark. Sequences generated by SA have structure-model and language-model summaries similar to those of stored sequences. Conditioning on the K/R-positive subset yields complete K/R marker retention; this is not an experimental binding measurement.

			AlphaFold2-ptm		ESMFold
Family	Source	$n$	pLDDT	TM	pLDDT	TM
Kunitz	Stored (natural)	50	$93.5\pm 4.0$	$0.83\pm 0.03$	$89.3\pm 3.2$	$0.83\pm 0.02$
	SA (K/R-positive)	50	$94.6\pm 1.4$	$0.84\pm 0.01$	$90.7\pm 1.1$	$0.83\pm 0.01$
	SA (full family)	50	$95.0\pm 1.3$	$0.85\pm 0.01$	$90.4\pm 1.2$	$0.84\pm 0.01$
$\omega$ -Conotoxin	Stored (natural)	50	$67.9\pm 9.8$	$0.35\pm 0.11$	$76.8\pm 7.7$	$0.43\pm 0.07$
	SA (designated subset)	50	$79.5\pm 4.0$	$0.47\pm 0.02$	$78.1\pm 2.5$	$0.47\pm 0.03$

Table 6: Cross-predictor structure-model comparison for Kunitz and

\omega

-conotoxin. Two structure predictors scored the same 50 sequences per source: AlphaFold2-ptm via ColabFold (1 model, 3 recycles, MMseqs2 MSA) and ESMFold. pLDDT: mean predicted confidence (

0

–

100

;

>70

= confident). TM: TM-score against the experimental reference (1BPI chain A for Kunitz, 1OMG chain A for

\omega

-conotoxin;

>0.5

is a conventional same-fold reference). Values are mean

\pm

s.d. across the 50 sequences. The designated-subset

\omega

-conotoxins had a higher mean TM-score than the stored group under both predictors. For full-family generation, the ESMFold mean was higher, whereas the AlphaFold2-ptm mean exceeded the stored mean by only 0.0027, negligible relative to the within-group standard deviations. For

\omega

-conotoxin, the overlap between stored and generated groups gives no evidence of a conditioning-induced loss. The low absolute TM-scores alone cannot distinguish short-chain metric behavior from structural differences. These model-derived quantities do not establish experimental folding or binding.

Condition	$N$	Tyr13 (%)	K+R (%)	KL	Novelty
Input: full family	74	33.8	9.9	0	0
Input: designated subset	23	82.6	19.7	0	0
Generated: full-seeded	1550	48.5	12.0	0.0084	0.59
Generated: designated-seeded	1550	98.5	20.1	0.0084	0.45

Table 7:

\omega

-Conotoxin Tyr13 marker retention under curated vs. full-family seeding. Tyr13%: fraction with tyrosine at the selected marker position (col. 13). K+R%: overall basic-residue fraction. KL: amino acid composition divergence. Novelty: mean cosine distance to nearest stored pattern. Designated-subset seeding increased the Tyr13 frequency from 82.6% to 98.5% and produced a similar basic-residue fraction (19.7% against 20.1%). Full-family seeding produced lower values than designated-subset seeding, although both Tyr13 and the basic-residue fraction increased relative to the full-family input.

Pos.	WT	Reported role	Evidence	Input	SA des.	SA full
13	Tyr	Reported Tyr marker	Ala: large activity loss	0.83	0.98	0.48
2	Lys	Loop 2 stabilization	Ala: 40 $\times$ loss (GVIA)	0.96	0.98	0.45
10	Arg	Reported loop 2 residue	MVIIA binding region	0.70	0.83	0.36
11	Leu	Reported loop 2 residue	MVIIA binding region	0.17	0.20	0.13
1	Cys	Cysteine framework	Disulfide-forming	1.00	1.00	1.00
8	Cys	Cysteine framework	Disulfide-forming	1.00	1.00	1.00

Table 8: Residue frequencies at reported

\omega

-conotoxin SAR positions. Published SAR and structural studies across

\omega

-conotoxins report roles for these positions in activity, binding, or fold maintenance [ 12 , 13 , 23 , 24 , 14 , 25 ] . For each reported position, the table gives the fraction retaining the listed residue (or a K/R substitution at basic positions). The six framework cysteines are invariant across the designated input, so the table reports their retention rather than their enrichment. Designated-subset generation retains the Tyr13 signal and the cysteine scaffold, and increases the listed frequencies at Tyr13, Arg10, position 21, and Lys4 relative to the designated input. These frequencies do not test the effects of mutations in the generated backgrounds.

Figure 1: Sequence properties vary with the number of designated inputs. Subsampling

K_{\mathrm{des}}\in\{3,5,8,10,15,20,25,32\}

sequences from the Kunitz K/R-positive set (3 replicates each; shaded bands show

\pm 1

s.d.). P1 K/R fraction was 1.0 at every sample size (not shown), so every input size retained the selected marker. Left: Pairwise sequence diversity increases with the number of designated inputs. Right: KL divergence of amino acid composition decreases over the same range.

Figure 2: Entropy crossover shifts rightward with increasing multiplicity ratio

\rho

. Attention entropy

H(\beta)

normalized by

\log K

(the uniform limit) as a function of inverse temperature

\beta

(log scale) for

\rho\in\{1,5,20,100,1000\}

on the Kunitz family (

K=99

K_{\mathrm{des}}=32

). At

\rho=1

(standard SA), the entropy starts at

H/\log K=1

(uniform attention over all patterns); as

\rho

increases, the multiplicity bias pre-concentrates the attention distribution, lowering the starting entropy toward

\log K_{\mathrm{des}}/\log K\approx 0.75

(dotted line). Each curve shows a sigmoidal drop to near-zero entropy; dots and dashed vertical lines mark the crossover onset

\beta^{*}(\rho)

, the point of maximum downward curvature. As

\rho

increases from 1 to 1000,

K_{\mathrm{eff}}

decreases from 99 to 32.1 and

\beta^{*}

shifts rightward from 4.4 to 9.3.

Figure 3: Decoded sequence readouts vary with PCA-space separation. (A) Observed readout fraction

f_{\mathrm{obs}}

versus effective designated fraction

f_{\mathrm{eff}}

as the multiplicity ratio

\rho

increases from 1 to 500 for six protein families. The dashed diagonal marks perfect calibration (

f_{\mathrm{obs}}=f_{\mathrm{eff}}

). Homeobox (

S=0.42

) and SH3 (

S=0.34

) domains track the diagonal closely; Kunitz (

S=0.20

) and Forkhead (

S=0.17

) curve below; WW domains (

S=0.11

) show the largest departure. The

\omega

-conotoxin curve compares accession-based designation with a Tyr13 readout rather than like quantities. For the five Pfam families, double-headed arrows indicate the calibration gap at

\rho=500

. (B) Separation index

S

versus calibration gap

\Delta

\rho=500

. Error bars show replicate s.d. An exploratory linear fit to the five Pfam families (filled markers;

\Delta\approx 0.73{}-1.8\,S

R^{2}=0.78

) summarizes the association without defining a prospective threshold. The

\omega

-conotoxin point (open diamond) is displayed as an external comparison and is not included in the fit; its vertical difference is descriptive, not a calibration error.

Figure 4: Recovery depends on retrieval sharpness and conditioning strength. P1 K/R fraction for the Kunitz designated subset versus inverse temperature

\beta

(log scale) under the hard attention mask (

b=-\infty

on background patterns; red curve, 30 chains per point, error bars are standard error across chains). Under the mask, all attention is restricted to designated memories. Recovery climbs monotonically from 0.47 at

\beta=2

to 1.00 at

\beta=512

with no plateau. The finite-

\rho

multiplicity conditions (blue diamonds, each at its own

\beta^{*}(\mathbf{r})

) sit below this envelope at matched

\beta

: hard masking recovers more marker signal than soft weighting by 0.085 (SE 0.028) at

f_{\mathrm{eff}}=0.5

, contracting into the noise as

\rho\to\infty

(0.003, SE 0.030, at

f_{\mathrm{eff}}=0.99

); faint red points show the mask run at each matched

\beta

. Because each condition’s attention already matches its own target (

\Delta_{\mathrm{attn}}\approx 0

), this shrinking margin tracks the difference in intended designated mass between the two conditions rather than a residual attention error. Hard curation (green triangle) reaches complete transfer at low

\beta

on its subset-only basis; unconditional generation (gray square) is shown for reference.

Figure 5: Binding-loop amino acid frequencies and generation residuals for the $\omega$ -conotoxin family (positions 9–17). Left column: Per-position amino acid frequency heatmaps for the input sequences (designated subset, $n=23$ , panel a; full O-superfamily, $n=74$ , panel c). Right column: Residual heatmaps (generated $-$ input) showing the change in frequency at each position-residue pair after SA generation. Panel (b): designated-seeded residual frequencies differ from the input by at most 0.16 in absolute value at every position-residue pair. The Tyr marker at position 13, the adjacent cysteines at positions 15 and 16, and the variable-loop diversity (positions 9–12) are reproduced within that bound. Panel (d): full-seeded residuals are slightly larger, reflecting the broader sequence diversity of the input. Blue: enrichment; red: depletion; white: no change.

Figure 6: Sequence-level analysis of SA-generated Kunitz domain sequences conditioned on the K/R-positive subset. (A) Alignment of the family consensus with the five highest-confidence SA K/R-positive-subset-conditioned sequences (ranked by ESMFold pLDDT). Positions are colored by conservation in the stored MSA: highly conserved (

>90\%

, red), conserved (

70

–

90\%

, orange), moderate (

50

–

70\%

, yellow), and variable (

<50\%

, blue). Cysteines forming the three disulfide bonds are highlighted in gold; the P1 position is highlighted in purple. Dots indicate matches to the consensus; letters indicate substitutions. All five sequences preserve every highly conserved position (

11/11

), all six cysteines (

6/6

), and carry K or R at P1, while introducing

19

–

26

substitutions concentrated at variable sites. (B) Per-position Shannon entropy for the stored MSA (gray), SA full-family generation (blue), and SA K/R-positive-subset-conditioned generation (red). Full-family generation tracks the stored entropy profile closely, showing similar position-specific conservation in the generated and stored sequences. The K/R-positive-subset profile tracks it slightly less closely, with the largest deviation at the P1 position (column 25): conditioning collapses P1 diversity to K/R, reducing entropy at this site while preserving the entropy profile elsewhere.

Figure 7: Sequence-level analysis of SA-generated

\omega

-conotoxin sequences conditioned on the designated subset. (A) Alignment of the family consensus with the five highest-confidence sequences among the 50 modeled SA designated-subset-conditioned sequences (ranked by ESMFold pLDDT). Positions are colored by conservation in the stored MSA: highly conserved (

>90\%

, red), conserved (

70

–

90\%

, orange), moderate (

50

–

70\%

, yellow), and variable (

<50\%

, blue). Cysteines forming the three disulfide bonds are highlighted in gold; the Tyr13 marker position is highlighted in green. All five sequences preserve all six highly conserved cysteine positions (

6/6

) and carry Tyr at the marker position, while introducing

3

–

9

substitutions concentrated at variable sites. (B) Per-position Shannon entropy for the stored MSA (gray), SA full-seeded generation (blue), and SA designated-seeded generation (red). Full-seeded generation tracks the stored entropy profile closely. The designated-seeded profile tracks it less closely, reflecting lower diversity after conditioning on the 23-sequence designated subset, including at Tyr13 and co-varying loop residues.

Figure 8: Structure models place selected SA-generated sequences near experimental family references. For each family, the single highest TM-score SA designated-subset-conditioned sequence (colored by per-residue pLDDT confidence) is superimposed onto the experimental reference structure (gray, semi-transparent) and shown in two orthogonal views (front and $90^{\circ}$ rotation) to reveal the full three-dimensional architecture. (A) Kunitz domain, front view. (B) Kunitz domain, $90^{\circ}$ rotation. The SA-generated variant aligns to the BPTI crystal structure (1BPI chain A) with TM $=0.86$ , reproducing the double-loop architecture, three-stranded antiparallel $\beta$ -sheet, and $\alpha$ -helical turn; mean C $\alpha$ confidence is 96.3. (C) $\omega$ -Conotoxin, front view. (D) $\omega$ -Conotoxin, $90^{\circ}$ rotation. The SA-generated variant aligns to the MVIIA NMR structure (1OMG chain A) with TM $=0.53$ ; its predicted backbone follows the reference in this illustrative alignment, with mean C $\alpha$ confidence 85.7. These highest-scoring examples are not group-level validation; group summaries appear in Table 6 . Colorbar: pLDDT confidence scale (red $=50$ , blue $=100$ ). Structures predicted by ESMFold [ 5 ] ; alignment by TM-align [ 21 ] .

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.323	$0.382\pm 0.031$	$0.327\pm 0.013$	$0.557\pm 0.004$
2	0.489	$0.412\pm 0.031$	$0.477\pm 0.009$	$0.556\pm 0.003$
5	0.705	$0.481\pm 0.014$	$0.712\pm 0.005$	$0.548\pm 0.004$
10	0.827	$0.515\pm 0.027$	$0.830\pm 0.005$	$0.538\pm 0.003$
20	0.905	$0.504\pm 0.015$	$0.904\pm 0.006$	$0.540\pm 0.001$
50	0.960	$0.554\pm 0.031$	$0.960\pm 0.005$	$0.531\pm 0.003$

Table S1 : Kunitz domain (PF00014)

\rho

sweep.

K=99

K_{\mathrm{des}}=32

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.600	$0.839\pm 0.021$	$0.602\pm 0.009$	$0.586\pm 0.004$
2	0.750	$0.861\pm 0.011$	$0.747\pm 0.005$	$0.582\pm 0.005$
5	0.882	$0.899\pm 0.013$	$0.889\pm 0.011$	$0.579\pm 0.002$
10	0.938	$0.900\pm 0.012$	$0.935\pm 0.003$	$0.581\pm 0.003$
20	0.968	$0.879\pm 0.020$	$0.967\pm 0.003$	$0.596\pm 0.008$
50	0.987	$0.969\pm 0.011$	$0.986\pm 0.002$	$0.539\pm 0.006$

Table S2 : SH3 domain (PF00018)

\rho

sweep.

K=55

K_{\mathrm{des}}=33

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.164	$0.144\pm 0.016$	$0.170\pm 0.006$	$0.686\pm 0.004$
2	0.282	$0.161\pm 0.009$	$0.288\pm 0.009$	$0.688\pm 0.005$
5	0.496	$0.187\pm 0.021$	$0.484\pm 0.015$	$0.686\pm 0.003$
10	0.663	$0.223\pm 0.028$	$0.659\pm 0.013$	$0.670\pm 0.007$
20	0.797	$0.255\pm 0.027$	$0.793\pm 0.007$	$0.671\pm 0.003$
50	0.908	$0.299\pm 0.025$	$0.911\pm 0.007$	$0.655\pm 0.003$

Table S3 : WW domain (PF00397)

\rho

sweep.

K=420

K_{\mathrm{des}}=69

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.750	$0.931\pm 0.017$	$0.747\pm 0.016$	$0.523\pm 0.005$
2	0.857	$0.939\pm 0.007$	$0.856\pm 0.013$	$0.518\pm 0.002$
5	0.938	$0.953\pm 0.008$	$0.936\pm 0.005$	$0.519\pm 0.003$
10	0.968	$0.948\pm 0.004$	$0.967\pm 0.001$	$0.520\pm 0.003$
20	0.984	$0.953\pm 0.014$	$0.984\pm 0.001$	$0.519\pm 0.005$
50	0.993	$0.954\pm 0.015$	$0.993\pm 0.001$	$0.519\pm 0.004$

Table S4 : Homeobox domain (PF00046)

\rho

sweep.

K=136

K_{\mathrm{des}}=102

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.496	$0.565\pm 0.013$	$0.496\pm 0.006$	$0.485\pm 0.003$
2	0.663	$0.590\pm 0.031$	$0.668\pm 0.009$	$0.485\pm 0.007$
5	0.831	$0.625\pm 0.025$	$0.830\pm 0.007$	$0.469\pm 0.004$
10	0.908	$0.612\pm 0.019$	$0.907\pm 0.005$	$0.468\pm 0.003$
20	0.952	$0.609\pm 0.030$	$0.951\pm 0.002$	$0.484\pm 0.003$
50	0.980	$0.688\pm 0.013$	$0.980\pm 0.002$	$0.434\pm 0.004$

Table S5 : Forkhead domain (PF00250)

\rho

sweep.

K=246

K_{\mathrm{des}}=122

$\rho$	$f_{\mathrm{eff}}$	$f_{\mathrm{obs}}$	$\bar{a}_{\mathrm{des}}$	$D$
1	0.311	$0.463\pm 0.040$	$0.304\pm 0.025$	$0.560\pm 0.003$
2	0.474	$0.530\pm 0.023$	$0.467\pm 0.010$	$0.550\pm 0.003$
5	0.693	$0.612\pm 0.022$	$0.667\pm 0.021$	$0.529\pm 0.009$
10	0.819	$0.662\pm 0.029$	$0.819\pm 0.013$	$0.510\pm 0.010$
20	0.900	$0.737\pm 0.025$	$0.899\pm 0.019$	$0.484\pm 0.007$
50	0.958	$0.798\pm 0.021$	$0.958\pm 0.004$	$0.464\pm 0.011$

Table S6 :

\omega

-conotoxin O-superfamily

\rho

sweep.

K=74

K_{\mathrm{des}}=23

$\rho$	$\beta/\beta^{*}$	$f_{\mathrm{obs}}$	$D$
10	0.50	0.463	0.571
10	1.00	0.487	0.537
10	1.50	0.515	0.510
10	2.00	0.534	0.496
10	3.00	0.558	0.482
50	0.50	0.485	0.566

Table S7 :

\beta

sweep on Kunitz domain (

K=99

K_{\mathrm{des}}=32

Family	$\rho$	$f_{\mathrm{eff}}$	SA marker	HMM marker	SA $D$	HMM $D$
Kunitz	1	0.323	$0.382\pm 0.031$	$0.306\pm 0.018$	0.557	0.634
Kunitz	10	0.827	$0.515\pm 0.027$	$0.820\pm 0.016$	0.538	0.606
Kunitz	100	0.979	$0.563\pm 0.023$	$0.980\pm 0.006$	0.519	0.589
Kunitz	500	0.996	$0.587\pm 0.029$	$0.996\pm 0.002$	0.510	0.590
SH3	1	0.600	$0.839\pm 0.021$	$0.545\pm 0.013$	0.586	0.711
SH3	10	0.938	$0.900\pm 0.012$	$0.928\pm 0.007$	0.581	0.665

Table S8 : Matched SA and sequence-weighted profile-HMM benchmark across representative multiplicity ratios. Both methods use the same cleaned alignments, designation labels, and relative multiplicity ratio. Marker values are mean

\pm

s.d. across five replicates of 620 sequences;

D

is mean pairwise sequence diversity.

Group	$n$	Mean pTM	Mean iPTM	Mean interface pLDDT
SA designated-subset	10	$0.221\pm 0.007$	$0.105\pm 0.008$	$37.2\pm 5.4$
SA full-family	10	$0.219\pm 0.008$	$0.104\pm 0.007$	$38.2\pm 4.0$
Natural controls	5	$0.228\pm 0.004$	$0.108\pm 0.007$	$40.7\pm 3.5$

Table S9 : Low-confidence AlphaFold2-multimer predictions for generated and control

\omega

-conotoxins. Values are group means

\pm

s.d.

Metric	SA des. vs SA full	SA des. vs controls	SA full vs controls
iPTM	1.0000	0.7502	0.4752
pTM	0.7873	0.1285	0.1092
Confidence	0.6560	0.1245	0.0726
Interface pLDDT	0.6556	0.2238	0.2934

Table S10 : Pairwise exact permutation p-values for

\omega

-conotoxin docking comparisons (all

H_{0}

: equal means).

$\rho$	$\beta^{*}$	$f_{\mathrm{eff}}$	MAP des., exact	MAP des., ULA	P1 K/R, exact	P1 K/R, ULA	AA KL
1	4.27	0.323	0.307	0.281	0.398	0.385	$1.40\times 10^{-4}$
10	4.93	0.827	0.894	0.887	0.515	0.499	$1.25\times 10^{-4}$
500	8.78	0.996	0.997	0.999	0.604	0.597	$1.19\times 10^{-4}$

Table S11 : Exact Gaussian-mixture versus finite-step ULA sampling on Kunitz. Exact and ULA columns use equal sample counts and the same

\beta^{*}

. The “MAP des.” column is the fraction assigned to a designated maximum-posterior component and is not the latent mixture weight

f_{\mathrm{eff}}

. The AA KL column is the amino acid composition divergence between the decoded exact and ULA libraries.

Figure S1 : Joint pLDDT and TM-score distributions for Kunitz sequences. Each point represents one predicted structure (50 per source). SA-generated (blue, red, green) and most stored (gray) sequences occupy the high-score region near pLDDT

>85

and TM-score

>0.8

. Profile HMM sequences (orange) scatter broadly, with many falling below the pLDDT

>70

confidence reference (vertical dashed line) and the conventional TM-score

>0.5

same-fold reference (horizontal dashed line).

Explore similar work

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention
Mar 16, 2026Jeffrey D. VarnerHopfield Networks Molecular Generation
R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850
Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
Mar 6, 2026Abdulrahman Alswaidan, Jeffrey D. VarnerHopfield Networks Energy-Based Models
R.F. Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850
Learning Latent Protein Languages for Autoregressive Generation
Oct 2, 2026Mahdi Pourmirzaei, Farzaneh Esmaili, Amir Ziashahabi +2Protein Language Models Generative Modeling
University of Missouri · University of Southern California · Independent Researcher

Mar 16, 2026J/K move · Enter open · S save

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

Jeffrey D. Varner

R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850

Hopfield Networks Molecular Generation Sequence Modeling

Open paper AskPDF arXiv

Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples. Yet most protein families are small: the median Pfam seed alignment contains only 22 sequences, a regime where learned models overfit or collapse. We propose stochastic attention (SA), a training-free sampler that treats the modern Hopfield energy over stored sequences as a Boltzmann distribution and draws samples via Langevin dynamics. The score function is the residual of a single softmax attention operation, eliminating the need for a trained score network, pretraining data, or graphics processing units (GPUs). Across eight Pfam families spanning 37 to 420 sequences and 23 to 262 residues, SA generates sequences with low composition divergence, novelty, and structural plausibility supported by ESMFold and AlphaFold2. Compared with profile hidden Markov models (HMMs), EvoDiff, and the multiple sequence alignment (MSA) Transformer, SA is the only tested method to simultaneously achieve low composition divergence, genuine novelty, and sequence identity within each family's nearest-neighbor identity range; the others drift outside this range or produce near-copies. The critical inverse temperature is predicted from principal component analysis (PCA) dimensionality alone, enabling fully automatic operation from a seed alignment. In two domains with deep mutational scanning data, SA-generated substitutions are enriched for experimentally tolerated mutations beyond a position-matched null, and an independent language model (ESM2-650M) scores them within the natural range. Stochastic attention thus opens training-free sequence generation to the long tail of protein families too small for deep learning.

Abstract

Explore similar work

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

Learning Latent Protein Languages for Autoregressive Generation