Attention heads retrieve: given a query, they return a weighted average of stored values. We showed that this computation is one step of gradient descent on the modern Hopfield energy, and that Langevin sampling from the corresponding Boltzmann distribution yielded stochastic attention, a training-free sampler controlled by a single temperature parameter. Lowering the temperature gave exact retrieval; raising it gave open-ended generation. Because the energy gradient equals the attention map, no score network, training loop, or learned model was required, making the approach particularly suited to the low-data regime where learned generative models are starved of training signal. We derived an entropy inflection condition that identified the retrieval-to-generation transition temperature for any memory geometry and validated the sampler on five domains spanning two orders of magnitude in dimension. A single Boolean mask on the attention softmax, identical to the causal mask used in transformers but applied along the memory axis rather than the sequence axis, turned the sampler into a zero-shot class-conditional generator on Olivetti faces with no retraining and no learned classifier. On MNIST digit images, stochastic attention produced samples that were markedly more novel and more diverse than the best learned baseline while matching a Metropolis-corrected gold standard. On protein sequences from a small Pfam family, the generation regime preserved amino acid composition far more faithfully than a variational autoencoder at matched novelty, indicating that the training-free score function retained family-level fidelity that learned models lost. A denoising diffusion baseline failed across all memory sizes tested, producing samples indistinguishable from isotropic noise. The approach required no architectural changes to the underlying attention mechanism.
Figures & tables
Figure 1: Synthetic experiments. (a) Phase behavior as a function of inverse temperature β ( d=64 , K=16 ). Left axis (blue): mean cosine similarity to the nearest stored pattern; right axis (coral): scaled entropy H(a)/logK . Both diagnostics reveal a smooth transition centered near β≈5–10 (gold band). (b) Convergence validation ( d=8 , K=4 , β=5 ). Pooled energy density from eight independent chains (gray) overlaid on a long-run reference distribution (coral); inset reports the Kolmogorov–Smirnov statistic and moment differences.
Method
N↑
Dˉ↑
Eˉ↓
Bootstrap (replay)
0.000±0.000
0.459±0.011
−0.500±0.000
Gaussian perturbation
0.004±0.000
0.450±0.013
−0.496±0.000
Random convex combination
0.092±0.000
0.008±0.000
−0.399±0.000
GMM-PCA ( r=50 , C=10 )
0.198±0.004
0.419±0.011
−0.303±0.005
VAE (latent =8 )
0.214±0.005
0.441±0.008
−0.286±0.005
DDPM ( T=200 )
0.938±0.002
0.991±0.002
+0.440±0.000 ‡
Table 1: Quantitative comparison on MNIST digit “3” ( K=100 , d=784 ). N and Dˉ are higher-is-better ( ↑ ); Eˉ is lower-is-better ( ↓ ). Values are mean ± SE across 30 chains. † Energy is positive at β=200 because samples explore off the attractor manifold. ‡ DDPM produces unstructured noise (max- cos=0.062 ).
Method
N↑
SeqID
KL ↓
Pos-KL ↓
MI r↑
HMM ↑
Bootstrap (replay)
0.000
0.644
0.143
7.52
0.692
100%
VAE (latent =8 )
0.621
0.532
0.416
9.99
0.525
100%
SA ( β=77 , retrieval)
0.243
0.616
0.107
5.66
0.733
100%
SA ( β=8 , generation)
0.623
0.538
0.060
2.92
0.871
100%
Table 2: Protein sequence generation on Pfam PF00076 ( K=68 , d=59 ). KL: global amino acid composition divergence. Pos-KL: mean per-position KL. MI r : Pearson correlation of pairwise mutual information matrices. HMM: fraction passing the Pfam RRM hidden Markov model (HMM) at E -value <0.01 .
Figure 2: Paired before/after grids on MNIST digit “3” at β=2000 . Each cell is one independent chain; cell (i,j) in panels (a, b, c) all share the same warm-start and the same seed. Compare cell-by-cell across panels, not within a row. (a) Stored “3” used as the warm-start. (b) SA chain endpoint after T=5,000 Langevin steps from that warm-start; digit identity is preserved while the textured background is the diffusive perturbation producing novelty (cf. Table 1 ). (c) MALA endpoint from the same warm-start and seed; visually identical to (b), so the Metropolis correction is unnecessary at α=0.01 . Bootstrap/Gaussian/Convex have no warm-start interpretation; see Table 1 for numerical comparison.
Figure 3: Paired before/after grids on Olivetti faces ( K=100 , d=4,096 ). Cell (i,j) in all three panels uses the same warm-start: the j -th stored portrait of subject i . (a) The 25 stored portraits used as warm-starts. (b) Hard-masked SA endpoints ( bk=−∞ outside the row’s subject); each row stays inside its subject (compare cell-by-cell to (a)). (c) Unconditional SA endpoints from the SAME warm-starts, mask removed; chains drift to other subjects within a row. Hard-mask subject-recovery: 96.0% vs. unconditional 10.4% vs. random-guess 10.0% .
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Phase diagram of attention concentration C=1−H(a)/logK over load ratio K/d (horizontal) and inverse temperature β (vertical, log scale), with d=64 . Each cell averages over five independent datasets. The dashed contour marks C=0.5 , separating a retrieval regime (upper-left, warm colors) from a diffuse regime (lower-right, dark colors).
Method
N↑
Dˉ↑
Eˉ↓
Bootstrap (replay)
0.000±0.000
0.416±0.019
−0.500±0.000
Gaussian perturbation
0.004±0.000
0.422±0.017
−0.496±0.000
Random convex combination
0.097±0.001
0.007±0.000
−0.398±0.001
GMM-PCA ( r=50 , C=10 )
0.130±0.005
0.401±0.017
−0.366±0.005
VAE (latent =8 )
0.180±0.006
0.408±0.008
−0.320±0.006
MALA
0.151±0.001
0.586±0.001
−0.305±0.001
Appendix
Table S1 : Quantitative comparison on MNIST digit “1” ( K=100 , d=784 , β=2000 ). Same protocol as Table 1 , including the VAE baseline (latent dim 8, same two-phase training). MALA acceptance rate: 99.2%. Values are mean ± SE across 30 chains.
Method
N↑
Dˉ↑
Eˉ↓
Bootstrap (replay)
0.000±0.000
0.421±0.012
−0.500±0.000
Gaussian perturbation
0.004±0.000
0.450±0.011
−0.496±0.000
Random convex combination
0.099±0.000
0.007±0.000
−0.395±0.000
GMM-PCA ( r=50 , C=10 )
0.194±0.005
0.420±0.011
−0.305±0.005
VAE (latent =8 )
0.209±0.005
0.397±0.009
−0.291±0.005
MALA
0.151±0.001
0.554±0.002
−0.305±0.001
Appendix
Table S2 : Quantitative comparison on MNIST digit “8” ( K=100 , d=784 , β=2000 ). Same protocol as Table 1 , including the VAE baseline (latent dim 8, same two-phase training). MALA acceptance rate: 99.2%. Values are mean ± SE across 30 chains.
Figure S2: Generated MNIST digit “1” samples ( 4×4 grids). The pattern matches digit “3”: only the Langevin-based methods (d, e) produce diverse, structured outputs.
Figure S3: Generated MNIST digit “8” samples ( 4×4 grids). Despite digit “8” having higher intra-class variance and more complex topology than digit “3”, the qualitative pattern is unchanged: Langevin methods dominate all baselines.
β
SNR
N↑
Dˉ↑
Eˉ↓
Mean max- cos
2000
0.113
0.152±0.001
0.557±0.001
−0.3±0.00
0.848±0.001
200
0.036
0.547±0.002
0.872±0.002
1.5±0.01
0.453±0.002
50
0.018
0.753±0.002
0.965±0.002
7.4±0.03
0.247±0.002
10
0.008
0.870±0.003
0.992±0.002
38.6±0.16
0.130±0.003
Appendix
Table S3 : Temperature spectrum for digit “8” ( K=100 , d=784 , α=0.01 , 30 chains). SNR=αβ/2d is the per-step signal-to-noise ratio; the transition occurs near SNR∗=α/(2d) (Proposition 3 ); at d=784 and α=0.01 this gives SNR∗≈0.013 . As SNR falls through this region the sampler crosses from structured retrieval into diffuse generation. Values are mean ± SE across 30 chains.
Figure S4: Temperature spectrum for MNIST digit “8.” From left to right: 16 stored patterns; SA samples at β=2000 (retrieval, local variation around stored patterns); β=200 (intermediate, genuine interpolation); β=50 (high novelty, structure fading); β=10 (diffuse noise). The trade-off between fidelity and novelty is governed entirely by β .
Protocol
β
SNR
N↑
Dˉ↑
Eˉ↓
Max- cos
Multi-chain (30 chains)
2000
0.113
0.152±0.001
0.600±0.001
−0.303±0.001
0.848±0.001
Single-chain
2000
0.113
0.153±0.000
0.282±0.001
−0.300±0.000
0.847±0.000
Single-chain
200
0.036
0.551±0.001
0.796±0.002
−1.50±0.00
0.449±0.001
Single-chain
50
0.018
0.756±0.002
0.967±0.002
−7.40±0.00
0.244±0.002
Appendix
Table S4 : Single-chain vs. multi-chain diversity on digit “3” ( K=100 , d=784 , α=0.01 ). SNR=αβ/2d . The multi-chain row reproduces the SA entry from Table 1 . The diversity gap of 0.318 at SNR=0.113 is the initialization contribution; single-chain diversity exceeds the multi-chain value once SNR falls toward the entropy inflection point (Proposition 3 ).
Figure S5 : Single-chain samples ( 6×6 grids, digit “3”) from a fixed seed at three operating points. (a) At SNR =0.113 the chain stays near one stored pattern; diversity is low ( 0.282 ). (b) At SNR =0.036 the chain spontaneously crosses energy barriers; single-chain diversity ( 0.796 ) exceeds the 30-chain β=2000 value ( 0.600 ), confirming genuine generation. (c) At SNR =0.018 the chain is fully diffuse; samples lose recognizable digit structure.
Figure S6: Chain-trajectory snapshots from a single Langevin chain at successive timesteps; no burn-in. Top: one MNIST chain at β=50 on digit “3”, warm-started from stored “3” #95; each cell is the denoise readout (one noise-free attention update at βread=2,000 ) of the chain state at the indicated T . The chain visits stored basins {95,95,100,18,53,96} across the six checkpoints, demonstrating inter-basin mixing on the ∼5,000 -step timescale established above. Bottom: one unconditional Olivetti chain at βchain=200 , warm-started in subject 0; each cell is the denoise readout at βread=10,000 . The chain leaves subject 0 by T=100 and visits subjects {6,7,4,6,8} at the subsequent checkpoints, illustrating the inter-basin drift the hard mask of the main text suppresses.
Method
N↑
Dˉ↑
Eˉ↓
SeqID
KL ↓
Bootstrap (replay)
0.000±0.000
1.000±0.012
−0.500±0.000
0.645±0.004
0.133
Gaussian perturbation
0.008±0.000
1.002±0.007
−0.492±0.000
0.633±0.006
0.132
Random convex combination
0.510±0.009
0.994±0.009
−0.068±0.001
0.562±0.000
2.091
GMM-PCA ( r=50 , C=10 )
0.606±0.009
1.000±0.007
+0.077±0.009
0.540±0.003
0.731
VAE (latent =8 )
0.621±0.007
0.834±0.022
+0.118±0.007
0.532±0.003
0.416
MALA ( β=77 )
0.249±0.004
0.992±0.007
−0.115±0.005
0.609±0.012
0.112
Appendix
Table S5 : Protein sequence generation on Pfam PF00076 (RRM family, K=68 , L=71 , d=59 after PCA). Two SA rows demonstrate the temperature knob: β=77 (retrieval, 20β∗ ) and β=8 (generation, 2β∗ ). SeqID: maximum sequence identity to any stored sequence. KL: amino acid composition divergence from stored sequences (lower is better). † Energy is positive at β=8 because samples explore off the attractor manifold. MALA acceptance rate: 99.8%.
Figure S7 : Protein sequence generation on PF00076. Left: Per-position amino acid frequencies for stored sequences (top) and SA-generated sequences (bottom). SA preserves the conserved positions of the RRM family while introducing variation at non-conserved sites. Right: Distribution of sequence identity to the nearest stored member for SA and GMM-PCA. SA generates sequences that are ≈62% identical to their nearest stored neighbor, while GMM-PCA produces more distant sequences ( ≈54% ) with much higher compositional divergence (KL =0.731 vs. 0.107 ).
Figure S8 : Attention entropy H(β)/logK for the Pfam RRM memory ( K=68 , d=59 ). The empirical inflection point β∗=3.8 (dashed blue) occurs at roughly half the random-pattern prediction d=7.7 (dotted green), reflecting the reduced effective similarity variance caused by conserved residues.
Figure S9 : QQ plots of generated (after per-ticker affine correction) vs. historical log returns for five representative S&P 500 constituents. Quantiles track closely for these tickers; 64.2% of all 424 tickers pass the two-sample KS test at the 5% level. The dashed red line is y=x .
Figure S10 : Pairwise correlation: historical vs. generated. Each point represents one of 89,676 asset pairs. The scatter around y=x shows partial preservation of cross-asset dependence structure (Frobenius error 26.3% vs. 15.6% for bootstrap).
Figure S11 : Empirical survival functions of equal-weighted portfolio returns. Both right tail P(g>x) and left tail P(g<x) match closely on a log scale.
Property
SA (corrected)
Bootstrap
Distributional fidelity (i.i.d. pool, 900 samples)
KS p -value (mean; 64.2% pass at 5% )
0.193
0.622
Cross-asset Frobenius error
26.3%
15.6%
Novelty ( 1−maxkcos )
0.768±0.001
0.000
Temporal properties (MALA warm-start, 2766 days)
SF2: Return unpredictability
Reproduced [ACF( g ) in 99% CI, β=12 ]
N/A
Appendix
Table S6 : Finance experiment: principled trade-offs of equilibrium Boltzmann sampling. SA and bootstrap occupy opposite ends of a novelty–fidelity spectrum: SA generates novel regime interpolations ( N=0.768±0.001 ) that bootstrap cannot produce, while bootstrap preserves marginals by construction. The Frobenius correlation error is identical before and after affine correction, confirming the energy captures dependence independently of scale. Temporal metrics apply to the sequential MALA experiment only.
Figure S12 : Stylized Facts 2 and 3: Autocorrelation analysis for five S&P 500 tickers. Top row : ACF( g ): the generated series (blue) stays within the 99% confidence band (dashed) at all lags, matching the near-zero autocorrelation of historical returns (red). Bottom row : ACF( g2 ): the historical series (red) shows persistent positive autocorrelation characteristic of volatility clustering, while the generated series (blue) shows ACF( g2 ) ≈0 , reflecting the absence of non-stationary dynamics in the equilibrium sampler.
Regime
Subject-recovery rate ↑
Random guess (uniform over 10 subjects)
0.100
Unconditional SA, uniform warm-start
0.104
Hard-masked SA ( bk=−∞ off-subject)
0.960
Appendix
Table S7 : Subject-recovery rate on Olivetti faces ( K=100 , d=4,096 , 10 subjects). Hard-masked SA recovers the target subject 9.2× more reliably than unconditional SA, matching random-guess. “Hard-mask” was computed across 5×5=25 samples spanning the first 5 subjects; “unconditional” across 50 chains warm-started uniformly across all 10 subjects, averaged against the same 5 target subjects.
β
2
8
32
128
512
2048
Novelty ↑
0.009
0.009
0.006
0.005
0.000
0.000
Hit rate ↑
1.000
1.000
0.950
0.900
0.900
0.900
Appendix
Table S8 : Single- β masked sampler on subject 0 of Olivetti, 20 chains per row. Novelty is the median minimum cosine distance to any in-subject stored portrait; hit rate is the nearest-mean classifier rate. The sweep recovers the modern-Hopfield → Hebbian transition empirically inside the masked regime.
α
Accept
Novelty
Diversity
ΔE
rate
ULA
MALA
ULA
MALA
(ULA − MALA)
0.001
> 0.99
0.151
0.150
0.595
0.595
< 0.001
0.005
> 0.99
0.151
0.151
0.599
0.597
< 0.001
0.01
0.992
0.152
0.151
0.600
0.598
0.002
0.02
0.978
0.153
0.151
0.600
0.598
0.003
0.05
0.911
0.155
0.151
0.602
0.598
0.006
Appendix
Table S9 : Step-size sweep on MNIST digit “3” ( K=100 , d=784 , β=2000 ). ULA and MALA are indistinguishable for α≤0.02 . At α≥0.2 , MALA acceptance collapses and the chain freezes, while ULA continues sampling.
Figure S13 : Step-size sweep: ULA vs. MALA on MNIST digit “3” ( β=2000 ). (a) MALA acceptance rate vs. step size α . Acceptance exceeds 95% for α≤0.02 and collapses to 0% at α≥0.2 . (b) Absolute difference between ULA and MALA metrics. All three metrics (novelty, diversity, energy) diverge smoothly as α increases through the [0.05,0.1] band.
Method
N↑
Dˉ↑
Eˉ↓
Max- cos↑
FD diag↓
Stored patterns
0.000
0.459
−0.50
1.000
—
SA ( β=200 )
0.547
0.885
+1.47
0.453
2.81
Gaussian (matched μ,σ2 )
0.672
0.878
+1.78
0.328
2.86
Gaussian (isotropic)
0.943
1.000
+2.34
0.057
3.91
Appendix
Table S10 : Gaussian noise control on MNIST digit “3” at β=200 (generation regime). Max- cos and energy cleanly separate SA from both Gaussian controls: SA samples retain structural similarity to stored patterns that noise cannot replicate. FD diag is the diagonal pixel-space Fréchet distance to stored patterns.
Figure S14: Gaussian noise control. (a) Stored digit “3” patterns. (b) SA at β=200 : noisy but spatially structured (curved strokes visible). (c) Matched Gaussian (same per-pixel μ and σ2 as SA): uniform static, no digit structure. (d) Isotropic Gaussian (matched norm): pure noise. (e) Distribution of max cosine similarity to nearest stored pattern: SA (blue) is clearly shifted right of both Gaussian controls, confirming that the Hopfield energy gradient preserves structural similarity that noise cannot replicate.
K
K/d
β∗
SA (generation, β=β∗ )
SA (retrieval, β=5β∗ )
DDPM ( T=200 )
N
Dˉ
max- cos
N
Dˉ
max- cos
max- cos
train (s)
100
0.13
9
0.874
0.993
0.126
0.763
0.969
0.237
0.061
34
500
0.64
13
0.851
0.991
0.149
0.722
0.955
0.278
0.073
37
1000
1.28
15
0.839
0.989
0.161
0.697
0.949
0.303
0.078
46
2000
2.55
18
0.827
0.987
0.173
0.673
0.940
0.327
0.086
66
3500
4.46
18
0.825
0.987
0.175
0.673
0.944
0.327
0.090
75
Appendix
Table S11 : Scaling experiment on MNIST digit “3”. β∗ is set via entropy inflection at each K . SA operates at β∗ (generation) and 5β∗ (retrieval). Max- cos measures structural similarity to stored patterns (isotropic noise baseline: ≈0.06 ).
Figure S15 : Scaling experiment: SA vs. DDPM on MNIST digit “3” as the number of stored patterns K increases from 100 to 3,500. (a) Max-cosine similarity: SA at both β∗ (generation) and 5β∗ (retrieval) produces samples with meaningful structural similarity to stored patterns that increases with K ; DDPM remains near the isotropic noise floor ( ≈0.06 ) at all K tested. (b) Novelty. (c) Diversity. (d) Entropy-inflection β∗ as a function of K .
β
Accept
NSA
ΔN
DˉSA
ΔDˉ
EˉSA
ΔEˉ
200
99.2%
0.548
+0.002
0.885
+0.002
+1.467
+0.018
500
99.2%
0.375
+0.002
0.782
+0.002
+0.287
+0.007
1000
99.2%
0.251
+0.002
0.687
+0.002
−0.107
+0.004
Appendix
Table S12 : ULA (SA) vs. MALA across the generation-to-retrieval spectrum. Δ columns report SA − MALA. All differences are negligible ( ∣Δ∣<0.02 ), confirming that ULA bias does not affect any experimental conclusion.
Method
Train data
MaxCos ↑
N↑
Dˉ↑
Eˉ↓
On-manif
VAE (lat =8 , original)
K=100
0.775
0.225
0.426
−0.276
100%
VAE-D3 (lat =8 )
N=1,000
0.668
0.332
0.603
−0.168
94%
VAE-D3 (lat =32 )
N=1,000
0.743
0.257
0.466
−0.244
100%
VAE-D3 (lat =64 )
N=1,000
0.762
0.238
0.413
−0.263
100%
VAE-all (lat =32 , top)
N=10,000
0.830
0.170
0.325
−0.330
100%
SA ( β=2000 )
K=100
0.848
0.152
0.600
−0.303
100%
Appendix
Table S13 : Comparison of SA against VAEs trained on 10× to 100× more data. All metrics computed against the same K=100 digit-3 memory matrix. MaxCos: mean max-cosine similarity to nearest stored pattern. On-manif: fraction of samples with Hopfield energy E<0 .
Config
K
N↑
Dˉ↑
MaxCos ↑
SA ( H=1 )
100
0.547±0.002
0.885±0.002
0.453±0.002
SA ( H=1 )
500
0.548±0.003
0.892±0.003
0.452±0.003
SA ( H=1 )
1000
0.548±0.002
0.892±0.003
0.452±0.002
SA ( H=2 , 50 PCs/head)
100
0.175±0.006
0.307±0.002
0.825±0.006
SA ( H=4 , 25 PCs/head)
100
0.187±0.005
0.291±0.002
0.815±0.005
SA ( H=5 , 20 PCs/head)
100
0.196±0.005
0.263±0.002
0.804±0.005
Appendix
Table S14 : Multi-head SA with PCA-partitioned subspaces versus single-head SA at larger memory sizes. All configurations at β=200 (generation regime). Metrics computed against each configuration’s own memory. MaxCos: mean maximum cosine similarity to the nearest stored pattern.
Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples. Yet most protein families are small: the median Pfam seed alignment contains only 22 sequences, a regime where learned models overfit or collapse. We propose stochastic attention (SA), a training-free sampler that treats the modern Hopfield energy over stored sequences as a Boltzmann distribution and draws samples via Langevin dynamics. The score function is the residual of a single softmax attention operation, eliminating the need for a trained score network, pretraining data, or graphics processing units (GPUs). Across eight Pfam families spanning 37 to 420 sequences and 23 to 262 residues, SA generates sequences with low composition divergence, novelty, and structural plausibility supported by ESMFold and AlphaFold2. Compared with profile hidden Markov models (HMMs), EvoDiff, and the multiple sequence alignment (MSA) Transformer, SA is the only tested method to simultaneously achieve low composition divergence, genuine novelty, and sequence identity within each family's nearest-neighbor identity range; the others drift outside this range or produce near-copies. The critical inverse temperature is predicted from principal component analysis (PCA) dimensionality alone, enabling fully automatic operation from a seed alignment. In two domains with deep mutational scanning data, SA-generated substitutions are enriched for experimentally tolerated mutations beyond a position-matched null, and an independent language model (ESM2-650M) scores them within the natural range. Stochastic attention thus opens training-free sequence generation to the long tail of protein families too small for deep learning.
Jeffrey D. Varner
R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850
Small protein-family alignments often contain a subset of interest but not enough labeled data to train a conditional generator. We condition a training-free stochastic-attention sampler by adding one multiplicity ratio to its logits. Increasing this ratio shifts generation from the full family toward the designated subset. For unit-norm memories, the resulting Boltzmann distribution is exactly a Gaussian mixture whose component weights are set by the multiplicities. This result separates exact conditioning in latent space from losses caused by sampling, PCA reconstruction, and sequence decoding. Across five Pfam families, attention followed the analytic target, but recovery of single-residue markers depended on how well PCA separated the designated and background sequences. A matched weighted profile HMM reproduced these markers more directly, while stochastic attention gave lower ESM2 pseudo-perplexity in the Kunitz comparison. Using a curated set of 23 omega-conotoxin sequences as the target subset produced diverse sequences that preserved the cysteine scaffold and Tyr13 and shifted other residues toward the designated set. These sequences are candidates for experimental testing; they do not establish binding.
Jeffrey D. Varner
Robert Frederick Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850
Auto-regressive models (ARMs) have established a dominant paradigm in language modeling. However, their strictly sequential sampling paradigm imposes fundamental constraints on both inference efficiency and modeling flexibility. To address these limitations, diffusion-based large language models (dLLMs) have been proposed, offering the potential for parallel sampling and flexible language modeling. Despite these advantages, current dLLMs sampling strategies rely primarily on token level information, which fails to account for global sequence structure and often yields suboptimal results. In this paper, we study the sampling order selection problem from the perspective of log-likelihood maximization. We show that this problem is NP-hard and propose an optimal sampling-rank-based approximation that makes the objective computationally tractable. We further prove that the tractable objective is optimized by sampling tokens in descending order of their attention-matrix column sums. This finding provides a principled justification for attention-guided sampling and offers a theoretically grounded alternative to greedy search. We instantiate this theoretical insight in a new training-free sampling algorithm, termed Attn-Sampler, and further propose dynamic attention thresholding for practical acceleration. Extensive experiments across multiple benchmarks validate the effectiveness of our proposed method, demonstrating that it achieves superior generation quality while enhancing the sampling parallelism.
Yuyan Zhou, Kai Syun Hou, Weiyu Chen +1
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology