Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

Abdulrahman Alswaidan; Jeffrey D. Varner

cs.LGMar 6, 2026

Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

Authors: Abdulrahman Alswaidan, Jeffrey D. Varner

Organizations: R.F. Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850

Abstract

Attention heads retrieve: given a query, they return a weighted average of stored values. We showed that this computation is one step of gradient descent on the modern Hopfield energy, and that Langevin sampling from the corresponding Boltzmann distribution yielded stochastic attention, a training-free sampler controlled by a single temperature parameter. Lowering the temperature gave exact retrieval; raising it gave open-ended generation. Because the energy gradient equals the attention map, no score network, training loop, or learned model was required, making the approach particularly suited to the low-data regime where learned generative models are starved of training signal. We derived an entropy inflection condition that identified the retrieval-to-generation transition temperature for any memory geometry and validated the sampler on five domains spanning two orders of magnitude in dimension. A single Boolean mask on the attention softmax, identical to the causal mask used in transformers but applied along the memory axis rather than the sequence axis, turned the sampler into a zero-shot class-conditional generator on Olivetti faces with no retraining and no learned classifier. On MNIST digit images, stochastic attention produced samples that were markedly more novel and more diverse than the best learned baseline while matching a Metropolis-corrected gold standard. On protein sequences from a small Pfam family, the generation regime preserved amino acid composition far more faithfully than a variational autoencoder at matched novelty, indicating that the training-free score function retained family-level fidelity that learned models lost. A denoising diffusion baseline failed across all memory sizes tested, producing samples indistinguishable from isotropic noise. The approach required no architectural changes to the underlying attention mechanism.

Figures & tables

Figure 1: Synthetic experiments. (a) Phase behavior as a function of inverse temperature

\beta

(

d=64

K=16

). Left axis (blue): mean cosine similarity to the nearest stored pattern; right axis (coral): scaled entropy

H(\mathbf{a})/\log K

. Both diagnostics reveal a smooth transition centered near

\beta\approx 5\text{--}10

(gold band). (b) Convergence validation (

d=8

K=4

\beta=5

). Pooled energy density from eight independent chains (gray) overlaid on a long-run reference distribution (coral); inset reports the Kolmogorov–Smirnov statistic and moment differences.

Method	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$
Bootstrap (replay)	$0.000\pm 0.000$	$0.459\pm 0.011$	$-0.500\pm 0.000$
Gaussian perturbation	$0.004\pm 0.000$	$0.450\pm 0.013$	$-0.496\pm 0.000$
Random convex combination	$0.092\pm 0.000$	$0.008\pm 0.000$	$-0.399\pm 0.000$
GMM-PCA ( $r{=}50$ , $C{=}10$ )	$0.198\pm 0.004$	$0.419\pm 0.011$	$-0.303\pm 0.005$
VAE (latent ${=}8$ )	$0.214\pm 0.005$	$0.441\pm 0.008$	$-0.286\pm 0.005$
DDPM ( $T{=}200$ )	$0.938\pm 0.002$	$0.991\pm 0.002$	$+0.44\phantom{0}\pm 0.00\phantom{0}$ ‡

Table 1: Quantitative comparison on MNIST digit “3” (

K=100

d=784

\mathcal{N}

and

\bar{\mathcal{D}}

are higher-is-better (

\uparrow

);

\bar{E}

is lower-is-better (

\downarrow

). Values are mean

\pm

SE across 30 chains. † Energy is positive at

\beta{=}200

because samples explore off the attractor manifold. ‡ DDPM produces unstructured noise (max-

\cos=0.062

Method	$\mathcal{N}$ $\uparrow$	SeqID	KL $\downarrow$	Pos-KL $\downarrow$	MI $r$ $\uparrow$	HMM $\uparrow$
Bootstrap (replay)	$0.000$	$0.644$	$0.143$	$7.52$	$0.692$	$100\%$
VAE (latent ${=}8$ )	$0.621$	$0.532$	$0.416$	$9.99$	$0.525$	$100\%$
SA ( $\beta{=}77$ , retrieval)	$0.243$	$0.616$	$0.107$	$5.66$	$0.733$	$100\%$
SA ( $\beta{=}8$ , generation)	$\mathbf{0.623}$	$0.538$	$\mathbf{0.060}$	$\mathbf{2.92}$	$\mathbf{0.871}$	$100\%$

Table 2: Protein sequence generation on Pfam PF00076 (

K{=}68

d{=}59

). KL: global amino acid composition divergence. Pos-KL: mean per-position KL. MI

r

: Pearson correlation of pairwise mutual information matrices. HMM: fraction passing the Pfam RRM hidden Markov model (HMM) at

E

-value

<0.01

Panel of Figure 2 — Figure 2: Paired before/after grids on MNIST digit “3” at $\beta{=}2000$ . Each cell is one independent chain; cell $(i,j)$ in panels (a, b, c) all share the same warm-start and the same seed. Compare cell-by-cell across panels, not within a row. (a) Stored “3” used as the warm-start. (b) SA chain endpoint after $T{=}5{,}000$ Langevin steps from that warm-start; digit identity is preserved while the textured background is the diffusive perturbation producing novelty (cf. Table 1 ). (c) MALA endpoint from the same warm-start and seed; visually identical to (b), so the Metropolis correction is unnecessary at $\alpha{=}0.01$ . Bootstrap/Gaussian/Convex have no warm-start interpretation; see Table 1 for numerical comparison.

Panel of Figure 3 — Figure 3: Paired before/after grids on Olivetti faces ( $K{=}100$ , $d{=}4{,}096$ ). Cell $(i,j)$ in all three panels uses the same warm-start: the $j$ -th stored portrait of subject $i$ . (a) The 25 stored portraits used as warm-starts. (b) Hard-masked SA endpoints ( $b_{k}{=}-\infty$ outside the row’s subject); each row stays inside its subject (compare cell-by-cell to (a)). (c) Unconditional SA endpoints from the SAME warm-starts, mask removed; chains drift to other subjects within a row. Hard-mask subject-recovery: $96.0\%$ vs. unconditional $10.4\%$ vs. random-guess $10.0\%$ .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Phase diagram of attention concentration $C=1-H(\mathbf{a})/\log K$ over load ratio $K/d$ (horizontal) and inverse temperature $\beta$ (vertical, log scale), with $d=64$ . Each cell averages over five independent datasets. The dashed contour marks $C=0.5$ , separating a retrieval regime (upper-left, warm colors) from a diffuse regime (lower-right, dark colors).

Method	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$
Bootstrap (replay)	$0.000\pm 0.000$	$0.416\pm 0.019$	$-0.500\pm 0.000$
Gaussian perturbation	$0.004\pm 0.000$	$0.422\pm 0.017$	$-0.496\pm 0.000$
Random convex combination	$0.097\pm 0.001$	$0.007\pm 0.000$	$-0.398\pm 0.001$
GMM-PCA ( $r{=}50$ , $C{=}10$ )	$0.130\pm 0.005$	$0.401\pm 0.017$	$-0.366\pm 0.005$
VAE (latent ${=}8$ )	$0.180\pm 0.006$	$0.408\pm 0.008$	$-0.320\pm 0.006$
MALA	$0.151\pm 0.001$	$0.586\pm 0.001$	$-0.305\pm 0.001$

Appendix

Table S1 : Quantitative comparison on MNIST digit “1” (

K{=}100

d{=}784

\beta{=}2000

). Same protocol as Table 1 , including the VAE baseline (latent dim 8, same two-phase training). MALA acceptance rate: 99.2%. Values are mean

\pm

SE across 30 chains.

Method	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$
Bootstrap (replay)	$0.000\pm 0.000$	$0.421\pm 0.012$	$-0.500\pm 0.000$
Gaussian perturbation	$0.004\pm 0.000$	$0.450\pm 0.011$	$-0.496\pm 0.000$
Random convex combination	$0.099\pm 0.000$	$0.007\pm 0.000$	$-0.395\pm 0.000$
GMM-PCA ( $r{=}50$ , $C{=}10$ )	$0.194\pm 0.005$	$0.420\pm 0.011$	$-0.305\pm 0.005$
VAE (latent ${=}8$ )	$0.209\pm 0.005$	$0.397\pm 0.009$	$-0.291\pm 0.005$
MALA	$0.151\pm 0.001$	$0.554\pm 0.002$	$-0.305\pm 0.001$

Appendix

Table S2 : Quantitative comparison on MNIST digit “8” (

K{=}100

d{=}784

\beta{=}2000

). Same protocol as Table 1 , including the VAE baseline (latent dim 8, same two-phase training). MALA acceptance rate: 99.2%. Values are mean

\pm

SE across 30 chains.

Panel of Figure 9 — Figure S2: Generated MNIST digit “1” samples ( $4{\times}4$ grids). The pattern matches digit “3”: only the Langevin-based methods (d, e) produce diverse, structured outputs.

Panel of Figure 10 — Figure S3: Generated MNIST digit “8” samples ( $4{\times}4$ grids). Despite digit “8” having higher intra-class variance and more complex topology than digit “3”, the qualitative pattern is unchanged: Langevin methods dominate all baselines.

$\beta$	SNR	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$	Mean max- $\cos$
2000	0.113	$0.152\pm 0.001$	$0.557\pm 0.001$	$-0.3\pm 0.00$	$0.848\pm 0.001$
200	0.036	$0.547\pm 0.002$	$0.872\pm 0.002$	$1.5\pm 0.01$	$0.453\pm 0.002$
50	0.018	$0.753\pm 0.002$	$0.965\pm 0.002$	$7.4\pm 0.03$	$0.247\pm 0.002$
10	0.008	$0.870\pm 0.003$	$0.992\pm 0.002$	$38.6\pm 0.16$	$0.130\pm 0.003$

Appendix

Table S3 : Temperature spectrum for digit “8” (

K{=}100

d{=}784

\alpha{=}0.01

, 30 chains).

\mathrm{SNR}=\sqrt{\alpha\beta/2d}

is the per-step signal-to-noise ratio; the transition occurs near

\mathrm{SNR}^{*}=\sqrt{\alpha/(2\sqrt{d})}

(Proposition 3 ); at

d{=}784

and

\alpha{=}0.01

this gives

\mathrm{SNR}^{*}\approx 0.013

. As SNR falls through this region the sampler crosses from structured retrieval into diffuse generation. Values are mean

\pm

SE across 30 chains.

Panel of Figure 12 — Figure S4: Temperature spectrum for MNIST digit “8.” From left to right: 16 stored patterns; SA samples at $\beta{=}2000$ (retrieval, local variation around stored patterns); $\beta{=}200$ (intermediate, genuine interpolation); $\beta{=}50$ (high novelty, structure fading); $\beta{=}10$ (diffuse noise). The trade-off between fidelity and novelty is governed entirely by $\beta$ .

Protocol	$\beta$	SNR	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$	Max- $\cos$
Multi-chain (30 chains)	2000	0.113	$0.152\pm 0.001$	$0.600\pm 0.001$	$-0.303\pm 0.001$	$0.848\pm 0.001$
Single-chain	2000	0.113	$0.153\pm 0.000$	$0.282\pm 0.001$	$-0.300\pm 0.000$	$0.847\pm 0.000$
Single-chain	200	0.036	$0.551\pm 0.001$	$0.796\pm 0.002$	$\phantom{-}1.5\phantom{0}\pm 0.0\phantom{0}$	$0.449\pm 0.001$
Single-chain	50	0.018	$0.756\pm 0.002$	$0.967\pm 0.002$	$\phantom{-}7.4\phantom{0}\pm 0.0\phantom{0}$	$0.244\pm 0.002$

Appendix

Table S4 : Single-chain vs. multi-chain diversity on digit “3” (

K{=}100

d{=}784

\alpha{=}0.01

\mathrm{SNR}=\sqrt{\alpha\beta/2d}

. The multi-chain row reproduces the SA entry from Table 1 . The diversity gap of

0.318

\mathrm{SNR}{=}0.113

is the initialization contribution; single-chain diversity exceeds the multi-chain value once SNR falls toward the entropy inflection point (Proposition 3 ).

Panel of Figure 14 — Figure S5 : Single-chain samples ( $6{\times}6$ grids, digit “3”) from a fixed seed at three operating points. (a) At SNR $=0.113$ the chain stays near one stored pattern; diversity is low ( $0.282$ ). (b) At SNR $=0.036$ the chain spontaneously crosses energy barriers; single-chain diversity ( $0.796$ ) exceeds the 30-chain $\beta{=}2000$ value ( $0.600$ ), confirming genuine generation. (c) At SNR $=0.018$ the chain is fully diffuse; samples lose recognizable digit structure.

Panel of Figure 15 — Figure S6: Chain-trajectory snapshots from a single Langevin chain at successive timesteps; no burn-in. Top: one MNIST chain at $\beta{=}50$ on digit “3”, warm-started from stored “3” #95; each cell is the denoise readout (one noise-free attention update at $\beta_{\mathrm{read}}{=}2{,}000$ ) of the chain state at the indicated $T$ . The chain visits stored basins $\{95,\,95,\,100,\,18,\,53,\,96\}$ across the six checkpoints, demonstrating inter-basin mixing on the $\sim 5{,}000$ -step timescale established above. Bottom: one unconditional Olivetti chain at $\beta_{\mathrm{chain}}{=}200$ , warm-started in subject 0; each cell is the denoise readout at $\beta_{\mathrm{read}}{=}10{,}000$ . The chain leaves subject 0 by $T{=}100$ and visits subjects $\{6,\,7,\,4,\,6,\,8\}$ at the subsequent checkpoints, illustrating the inter-basin drift the hard mask of the main text suppresses.

Method	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$	SeqID	KL $\downarrow$
Bootstrap (replay)	$0.000\pm 0.000$	$1.000\pm 0.012$	$-0.500\pm 0.000$	$0.645\pm 0.004$	0.133
Gaussian perturbation	$0.008\pm 0.000$	$1.002\pm 0.007$	$-0.492\pm 0.000$	$0.633\pm 0.006$	0.132
Random convex combination	$0.510\pm 0.009$	$0.994\pm 0.009$	$-0.068\pm 0.001$	$0.562\pm 0.000$	2.091
GMM-PCA ( $r{=}50$ , $C{=}10$ )	$0.606\pm 0.009$	$1.000\pm 0.007$	$+0.077\pm 0.009$	$0.540\pm 0.003$	0.731
VAE (latent ${=}8$ )	$0.621\pm 0.007$	$0.834\pm 0.022$	$+0.118\pm 0.007$	$0.532\pm 0.003$	0.416
MALA ( $\beta{=}77$ )	$0.249\pm 0.004$	$0.992\pm 0.007$	$-0.115\pm 0.005$	$0.609\pm 0.012$	0.112

Appendix

Table S5 : Protein sequence generation on Pfam PF00076 (RRM family,

K{=}68

L{=}71

d{=}59

after PCA). Two SA rows demonstrate the temperature knob:

\beta{=}77

(retrieval,

20\beta^{*}

) and

\beta{=}8

(generation,

2\beta^{*}

). SeqID: maximum sequence identity to any stored sequence. KL: amino acid composition divergence from stored sequences (lower is better). † Energy is positive at

\beta{=}8

because samples explore off the attractor manifold. MALA acceptance rate: 99.8%.

Panel of Figure 17 — Figure S7 : Protein sequence generation on PF00076. Left: Per-position amino acid frequencies for stored sequences (top) and SA-generated sequences (bottom). SA preserves the conserved positions of the RRM family while introducing variation at non-conserved sites. Right: Distribution of sequence identity to the nearest stored member for SA and GMM-PCA. SA generates sequences that are ${\approx}62\%$ identical to their nearest stored neighbor, while GMM-PCA produces more distant sequences ( ${\approx}54\%$ ) with much higher compositional divergence (KL $=0.731$ vs. $0.107$ ).

Figure S8 : Attention entropy

H(\beta)/\log K

for the Pfam RRM memory (

K{=}68

d{=}59

). The empirical inflection point

\beta^{*}{=}3.8

(dashed blue) occurs at roughly half the random-pattern prediction

\sqrt{d}{=}7.7

(dotted green), reflecting the reduced effective similarity variance caused by conserved residues.

Figure S9 : QQ plots of generated (after per-ticker affine correction) vs. historical log returns for five representative S&P 500 constituents. Quantiles track closely for these tickers;

64.2\%

of all

424

tickers pass the two-sample KS test at the

5\%

level. The dashed red line is

y=x

Figure S10 : Pairwise correlation: historical vs. generated. Each point represents one of $89{,}676$ asset pairs. The scatter around $y=x$ shows partial preservation of cross-asset dependence structure (Frobenius error $26.3\%$ vs. $15.6\%$ for bootstrap).

Figure S11 : Empirical survival functions of equal-weighted portfolio returns. Both right tail

P(g>x)

and left tail

P(g<x)

match closely on a log scale.

Property	SA (corrected)	Bootstrap
Distributional fidelity (i.i.d. pool, 900 samples)
KS $p$ -value (mean; 64.2% pass at $5\%$ )	0.193	0.622
Cross-asset Frobenius error	26.3%	15.6%
Novelty ( $1-\max_{k}\cos$ )	$0.768\pm 0.001$	0.000
Temporal properties (MALA warm-start, 2766 days)
SF2: Return unpredictability	Reproduced [ACF( $g$ ) in 99% CI, $\beta{=}12$ ]	N/A

Appendix

Table S6 : Finance experiment: principled trade-offs of equilibrium Boltzmann sampling. SA and bootstrap occupy opposite ends of a novelty–fidelity spectrum: SA generates novel regime interpolations (

\mathcal{N}=0.768\pm 0.001

) that bootstrap cannot produce, while bootstrap preserves marginals by construction. The Frobenius correlation error is identical before and after affine correction, confirming the energy captures dependence independently of scale. Temporal metrics apply to the sequential MALA experiment only.

Figure S12 : Stylized Facts 2 and 3: Autocorrelation analysis for five S&P 500 tickers. Top row : ACF(

g

): the generated series (blue) stays within the 99% confidence band (dashed) at all lags, matching the near-zero autocorrelation of historical returns (red). Bottom row : ACF(

g^{2}

): the historical series (red) shows persistent positive autocorrelation characteristic of volatility clustering, while the generated series (blue) shows ACF(

g^{2}

)

\approx 0

, reflecting the absence of non-stationary dynamics in the equilibrium sampler.

Regime	Subject-recovery rate $\uparrow$
Random guess (uniform over 10 subjects)	$0.100$
Unconditional SA, uniform warm-start	$0.104$
Hard-masked SA ( $b_{k}{=}-\infty$ off-subject)	$\mathbf{0.960}$

Appendix

Table S7 : Subject-recovery rate on Olivetti faces (

K{=}100

d{=}4{,}096

, 10 subjects). Hard-masked SA recovers the target subject

9.2\times

more reliably than unconditional SA, matching random-guess. “Hard-mask” was computed across

5\times 5=25

samples spanning the first 5 subjects; “unconditional” across

50

chains warm-started uniformly across all 10 subjects, averaged against the same 5 target subjects.

$\beta$	$2$	$8$	$32$	$128$	$512$	$2048$
Novelty $\uparrow$	$0.009$	$0.009$	$0.006$	$0.005$	$0.000$	$0.000$
Hit rate $\uparrow$	$1.000$	$1.000$	$0.950$	$0.900$	$0.900$	$0.900$

Appendix

Table S8 : Single-

\beta

masked sampler on subject 0 of Olivetti,

20

chains per row. Novelty is the median minimum cosine distance to any in-subject stored portrait; hit rate is the nearest-mean classifier rate. The sweep recovers the modern-Hopfield

\to

Hebbian transition empirically inside the masked regime.

$\alpha$	Accept	Novelty		Diversity		$\Delta E$
	rate	ULA	MALA	ULA	MALA	(ULA $-$ MALA)
0.001	$>$ 0.99	0.151	0.150	0.595	0.595	$<$ 0.001
0.005	$>$ 0.99	0.151	0.151	0.599	0.597	$<$ 0.001
0.01	0.992	0.152	0.151	0.600	0.598	0.002
0.02	0.978	0.153	0.151	0.600	0.598	0.003
0.05	0.911	0.155	0.151	0.602	0.598	0.006

Appendix

Table S9 : Step-size sweep on MNIST digit “3” (

K{=}100

d{=}784

\beta{=}2000

). ULA and MALA are indistinguishable for

\alpha\leq 0.02

. At

\alpha\geq 0.2

, MALA acceptance collapses and the chain freezes, while ULA continues sampling.

Figure S13 : Step-size sweep: ULA vs. MALA on MNIST digit “3” (

\beta{=}2000

). (a) MALA acceptance rate vs. step size

\alpha

. Acceptance exceeds

95\%

for

\alpha\leq 0.02

and collapses to

0\%

\alpha\geq 0.2

. (b) Absolute difference between ULA and MALA metrics. All three metrics (novelty, diversity, energy) diverge smoothly as

\alpha

increases through the

[0.05,0.1]

band.

Method	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$	Max- $\cos$ $\uparrow$	FD ${}_{\text{diag}}$ $\downarrow$
Stored patterns	0.000	0.459	$-0.50$	1.000	—
SA ( $\beta{=}200$ )	0.547	0.885	$+1.47$	0.453	2.81
Gaussian (matched $\mu,\sigma^{2}$ )	0.672	0.878	$+1.78$	0.328	2.86
Gaussian (isotropic)	0.943	1.000	$+2.34$	0.057	3.91

Appendix

Table S10 : Gaussian noise control on MNIST digit “3” at

\beta{=}200

(generation regime). Max-

\cos

and energy cleanly separate SA from both Gaussian controls: SA samples retain structural similarity to stored patterns that noise cannot replicate. FD

{}_{\text{diag}}

is the diagonal pixel-space Fréchet distance to stored patterns.

Panel of Figure 29 — Figure S14: Gaussian noise control. (a) Stored digit “3” patterns. (b) SA at $\beta{=}200$ : noisy but spatially structured (curved strokes visible). (c) Matched Gaussian (same per-pixel $\mu$ and $\sigma^{2}$ as SA): uniform static, no digit structure. (d) Isotropic Gaussian (matched norm): pure noise. (e) Distribution of max cosine similarity to nearest stored pattern: SA (blue) is clearly shifted right of both Gaussian controls, confirming that the Hopfield energy gradient preserves structural similarity that noise cannot replicate.

$K$	$K/d$	$\beta^{*}$	SA (generation, $\beta{=}\beta^{*}$ )			SA (retrieval, $\beta{=}5\beta^{*}$ )			DDPM ( $T{=}200$ )
			$\mathcal{N}$	$\bar{\mathcal{D}}$	max- $\cos$	$\mathcal{N}$	$\bar{\mathcal{D}}$	max- $\cos$	max- $\cos$	train (s)
100	0.13	9	0.874	0.993	0.126	0.763	0.969	0.237	0.061	34
500	0.64	13	0.851	0.991	0.149	0.722	0.955	0.278	0.073	37
1000	1.28	15	0.839	0.989	0.161	0.697	0.949	0.303	0.078	46
2000	2.55	18	0.827	0.987	0.173	0.673	0.940	0.327	0.086	66
3500	4.46	18	0.825	0.987	0.175	0.673	0.944	0.327	0.090	75

Appendix

Table S11 : Scaling experiment on MNIST digit “3”.

\beta^{*}

is set via entropy inflection at each

K

. SA operates at

\beta^{*}

(generation) and

5\beta^{*}

(retrieval). Max-

\cos

measures structural similarity to stored patterns (isotropic noise baseline:

{\approx}0.06

Figure S15 : Scaling experiment: SA vs. DDPM on MNIST digit “3” as the number of stored patterns

K

increases from 100 to 3,500. (a) Max-cosine similarity: SA at both

\beta^{*}

(generation) and

5\beta^{*}

(retrieval) produces samples with meaningful structural similarity to stored patterns that increases with

K

; DDPM remains near the isotropic noise floor (

{\approx}0.06

) at all

K

tested. (b) Novelty. (c) Diversity. (d) Entropy-inflection

\beta^{*}

as a function of

K

$\beta$	Accept	$\mathcal{N}_{\text{SA}}$	$\Delta\mathcal{N}$	$\bar{\mathcal{D}}_{\text{SA}}$	$\Delta\bar{\mathcal{D}}$	$\bar{E}_{\text{SA}}$	$\Delta\bar{E}$
200	99.2%	0.548	$+0.002$	0.885	$+0.002$	$+1.467$	$+0.018$
500	99.2%	0.375	$+0.002$	0.782	$+0.002$	$+0.287$	$+0.007$
1000	99.2%	0.251	$+0.002$	0.687	$+0.002$	$-0.107$	$+0.004$

Appendix

Table S12 : ULA (SA) vs. MALA across the generation-to-retrieval spectrum.

\Delta

columns report SA

-

MALA. All differences are negligible (

|\Delta|<0.02

), confirming that ULA bias does not affect any experimental conclusion.

Method	Train data	MaxCos $\uparrow$	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	$\bar{E}$ $\downarrow$	On-manif
VAE (lat ${=}8$ , original)	$K{=}100$	$0.775$	$0.225$	$0.426$	$-0.276$	$100\%$
VAE-D3 (lat ${=}8$ )	$N{=}1{,}000$	$0.668$	$0.332$	$0.603$	$-0.168$	$94\%$
VAE-D3 (lat ${=}32$ )	$N{=}1{,}000$	$0.743$	$0.257$	$0.466$	$-0.244$	$100\%$
VAE-D3 (lat ${=}64$ )	$N{=}1{,}000$	$0.762$	$0.238$	$0.413$	$-0.263$	$100\%$
VAE-all (lat ${=}32$ , top)	$N{=}10{,}000$	$0.830$	$0.170$	$0.325$	$-0.330$	$100\%$
SA ( $\beta{=}2000$ )	$K{=}100$	$\mathbf{0.848}$	$0.152$	$\mathbf{0.600}$	$-0.303$	$100\%$

Appendix

Table S13 : Comparison of SA against VAEs trained on

10{\times}

100{\times}

more data. All metrics computed against the same

K{=}100

digit-3 memory matrix. MaxCos: mean max-cosine similarity to nearest stored pattern. On-manif: fraction of samples with Hopfield energy

E<0

Config	$K$	$\mathcal{N}$ $\uparrow$	$\bar{\mathcal{D}}$ $\uparrow$	MaxCos $\uparrow$
SA ( $H{=}1$ )	100	$0.547\pm 0.002$	$0.885\pm 0.002$	$0.453\pm 0.002$
SA ( $H{=}1$ )	500	$0.548\pm 0.003$	$0.892\pm 0.003$	$0.452\pm 0.003$
SA ( $H{=}1$ )	1000	$0.548\pm 0.002$	$0.892\pm 0.003$	$0.452\pm 0.002$
SA ( $H{=}2$ , 50 PCs/head)	100	$0.175\pm 0.006$	$0.307\pm 0.002$	$0.825\pm 0.006$
SA ( $H{=}4$ , 25 PCs/head)	100	$0.187\pm 0.005$	$0.291\pm 0.002$	$0.815\pm 0.005$
SA ( $H{=}5$ , 20 PCs/head)	100	$0.196\pm 0.005$	$0.263\pm 0.002$	$0.804\pm 0.005$

Appendix

Table S14 : Multi-head SA with PCA-partitioned subspaces versus single-head SA at larger memory sizes. All configurations at

\beta{=}200

(generation regime). Metrics computed against each configuration’s own memory. MaxCos: mean maximum cosine similarity to the nearest stored pattern.

Explore similar work

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention
Mar 16, 2026Jeffrey D. VarnerHopfield Networks Molecular Generation
R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850
Conditioning Protein Generation via Hopfield Pattern Multiplicity
Mar 20, 2026Jeffrey D. VarnerConditional Generative Modeling Conditional Generation
Robert Frederick Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850
Attention-Based Sampler for Diffusion Language Models
Mar 18, 2026Yuyan Zhou, Kai Syun Hou, Weiyu Chen +1Diffusion Sampling Diffusion Language Model Decoding
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology

Mar 16, 2026J/K move · Enter open · S save

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

Jeffrey D. Varner

R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850

Hopfield Networks Molecular Generation Sequence Modeling

Open paper AskPDF arXiv

Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples. Yet most protein families are small: the median Pfam seed alignment contains only 22 sequences, a regime where learned models overfit or collapse. We propose stochastic attention (SA), a training-free sampler that treats the modern Hopfield energy over stored sequences as a Boltzmann distribution and draws samples via Langevin dynamics. The score function is the residual of a single softmax attention operation, eliminating the need for a trained score network, pretraining data, or graphics processing units (GPUs). Across eight Pfam families spanning 37 to 420 sequences and 23 to 262 residues, SA generates sequences with low composition divergence, novelty, and structural plausibility supported by ESMFold and AlphaFold2. Compared with profile hidden Markov models (HMMs), EvoDiff, and the multiple sequence alignment (MSA) Transformer, SA is the only tested method to simultaneously achieve low composition divergence, genuine novelty, and sequence identity within each family's nearest-neighbor identity range; the others drift outside this range or produce near-copies. The critical inverse temperature is predicted from principal component analysis (PCA) dimensionality alone, enabling fully automatic operation from a seed alignment. In two domains with deep mutational scanning data, SA-generated substitutions are enriched for experimentally tolerated mutations beyond a position-matched null, and an independent language model (ESM2-650M) scores them within the natural range. Stochastic attention thus opens training-free sequence generation to the long tail of protein families too small for deep learning.

Abstract

Explore similar work

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

Conditioning Protein Generation via Hopfield Pattern Multiplicity

Attention-Based Sampler for Diffusion Language Models