Dominant methods for interpreting large models impose strong structural assumptions. Generative Latent Priors (GLP) are a complementary approach: a diffusion model that learns the distribution of a frozen model's activations directly, imposing no assumptions. We port GLP to ESM-2 activations (UniRef50) and evaluate it at two host scales (ESM-2-8M and -650M), both as a lens for protein representations and as a prior for editing them. Generation quality scales smoothly with depth and compute. As a lens, single GLP "meta-neurons" are the strongest probes of discrete Swiss-Prot concepts at both hosts, beating InterPLM's released SAE features, and lead on continuous properties at 8M though not at 650M; falsification tests show this is mostly concentration, not new information. As an editing prior, the on-manifold correction makes DiffMean steering of three biochemical properties effective and restores on-manifoldness, but only in isolation: re-embedding the sequences actually produced after several rounds of decoding, the structural plausibility is compromised. This is mainly reproduced in additional SAE-based clamping structural steering experiments, where only Helix motif succeeds. The naturalness failure is traced to the rewriting coverage and the decoder rather than the prior, highlighting what the method must clear before it can serve for both discovery-by-editing and discovery-by-exploration. Trained GLPs are released on huggingface.co/collections/HPAI-BSC/glprot.
Figures & tables
Figure 1: Pipeline overview. (a) GLP trains by flow matching on ESM-2 activations from UniRef50 to learn their manifold. (b) Steering pushes an activation off that manifold toward a target property. (c) GLP correction re-noises and denoises the edit back onto the manifold.
Discrete AUC
Continuous ρ
Representation
8M
650M
8M
650M
GLP
0.903 ∗ [0.887, 0.918]
0.932 ∗ [0.917, 0.946]
0.669 ∗ [0.617, 0.725]
0.555 [0.498, 0.613]
SAE Simon and Zou (2025)
0.845 [0.826, 0.865]
0.872 [0.850, 0.893]
0.602 [0.557, 0.650]
0.569 [0.498, 0.648]
ESM FFN
0.877 [0.859, 0.894]
0.850 [0.829, 0.871]
0.563 [0.525, 0.600]
0.373 [0.325, 0.420]
Res. Stream
0.833 [0.815, 0.850]
0.872 [0.852, 0.891]
0.478 [0.413, 0.549]
0.373 [0.317, 0.424]
Table 1: 1-D probing: mean test AUC or Spearman ρ [95% bootstrap CI]. Bold: highest per column; ∗ : p<0.05 vs. all baselines (two-sided paired Wilcoxon).
Figure 2: Property steering naive vs GLP correction, both directions. Change vs. α=0
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Generation quality scales with compute. (a) Smoothed final loss of all eight (depth × data) runs collapses onto one power law in C=6ND ; our protein-activation curve lies below the published LLM-activation reference (dotted). (b) FD lower-bound ratio (FD@1000 / real-vs-real floor; lower is better) vs GLP depth, on matched 100M-activation grids
Data
Blocks
Params
C=6ND
FD@1000
FD ratio
Loss
ESM-2-8M host ( d=320 )
100M
3
11.1M
6.7×1015
4.27
4.47 ×
1.017
100M
6
20.9M
1.3×1016
4.28
4.47 ×
0.989
100M
12
40.6M
2.4×1016
3.97
4.15 ×
0.956
100M
24
80.0M
4.8×1016
3.88
4.06 ×
0.924
500M
3
11.1M
3.3×1016
3.54
3.33 ×
0.946
Appendix
Table 2: Depth × data-scale grid, generation quality only. N = parameters, C=6ND , FD at 1000 sampler steps; ratio is FD / real-vs-real lower bound; loss is the mean of the last 500 training steps. The two hosts occupy different-dimensional activation spaces ( d=320 vs. 1280 ), so absolute FD and loss are not comparable across host blocks; only the FD ratio is. Bold: the model used for all probing and steering results.
Figure 4: Fréchet Distance vs. sampler steps, by GLP depth and data scale (shared y -axis). FD falls with sampler steps everywhere. Depth lowers FD monotonically and the gain is larger at 500 M than at 100 M: more data lets extra depth pay off. The shallowest cells ( 3 vs. 6 blocks) are near-identical at both scales; separation appears at 12 – 24 blocks. Dashed lines: real-vs-real lower bound.
Figure 5: Sampler convergence in PCA space (24-block, 500M): generated activations (orange) refine from a collapsed blob at 1 step to matching the real distribution (purple) at 1000 steps, on the real data’s top-2 principal components. Inset: training loss, converging below the trivial-predictor floor ( ≈2.0 ).
property
α
naive
p
GLP-corrected
p
SD
gravy
0.032
−0.171
0.17
+0.248
0.02
+0.54
gravy
0.064
+0.006
0.93
+0.551
1×10−5
+1.21
gravy
0.128
+0.112
0.23
+0.969
7×10−10
+2.12
gravy
0.257
+0.077
0.44
+1.581
2×10−13
+3.46
gravy
0.385
+0.546
8×10−5
+2.507
4×10−14
+5.49
gravy
0.642
+1.400
1×10−10
+3.382
2×10−13
+7.41
Appendix
Table 3: Steering-attributable divergence (up − down); α in units of ∥h∥ , SD = divergence over the property’s UniRef50 spread. Unpaired Mann–Whitney, n=40 per direction.
naive
GLP-corrected
property
from low
from high
from low
from high
fixed bias (SD)
hydrophobicity
+0.347
−0.282
+1.145
+0.536
+1.77
charge
+0.055
−0.039
+0.063
−0.071
−0.20
instability
+14.99
−15.48
+4.66
−27.13
−0.70
Appendix
Table 4: Concept change with no steering applied ( α=0 ), by arm and starting set, in each property’s own units. A fixed bias adds the same amount from both sets; mean reversion flips sign with the set and cancels in the corrected − naive difference, which is the last column.
Figure 6: The α=0 arms, in units of each property’s spread. Hatched: naive. A bias that reverses with the starting set is mean reversion; one that does not comes from the correction.
Figure 7: GLP diffusion loss against steering strength, before and after a single correction pass applied directly to the steered activation (no generation), for hydrophobicity
starting set
natural pTM
after decoding
change
instability, stable
0.650
0.294
−55%
hydrophobicity, hydrophobic
0.629
0.315
−50%
charge, anionic
0.507
0.256
−50%
charge, cationic
0.438
0.257
−41%
hydrophobicity, hydrophilic
0.373
0.274
−27%
instability, unstable
0.369
0.227
−38%
Appendix
Table 5: Foldability before and after the decoding loop, with no steering and no correction ( α=0 , naive arm), for all six held-out sets.
Figure 8: Steering effect (top) and foldability (bottom) against how much of the protein the decoder rewrites, T=0.5 . Marker shape distinguishes mask fraction; the three series fall on a single curve, i.e. mask fraction and round count act only through their product. Shaded band: the starting proteins’ own pTM ( ± SEM). Grey points are settings whose output is mostly degenerate and are excluded from the trend. This budget sweep predates the relative- α parameterisation and its axis label is in raw units; for the hydrophobicity vector its α=2 is the same perturbation as the α=0.257 used elsewhere in this paper.
decoder
degenerate
pTM ( α=0 )
divergence
beam, E=5
2.5%
0.486
+0.07
beam, E=10
2.5%
0.473
+0.08
beam, E=25
3.5%
0.438
+0.27
beam, E=50
4.6%
0.409
+0.51
masked-Gibbs, 30 rounds
8.5%
0.293
+3.99
Appendix
Table 6: Beam search over the steered pseudo-likelihood as a function of the edit budget E , against uniform-random masked-Gibbs. pTM is the α=0 naive arm (the decoder alone); divergence is the attributable up-minus-down effect at the top of the grid; both at n=40 starting proteins per direction, hydrophobicity.
c
structure (naive)
structure (corr.)
pLDDT (naive/corr.)
pTM (naive/corr.)
Helix, feature 5355 (DSSP helix fraction)
0
0.54
0.77
49.7 / 59.8
0.153 / 0.251
1
0.73
0.78
59.7 / 62.3
0.270 / 0.284
2
0.74
0.80
61.5 / 63.5
0.287 / 0.296
3
0.76
0.77
61.8 / 61.7
0.290 / 0.285
4
0.76
0.78
61.6 / 62.4
0.287 / 0.296
Appendix
Table 7: Structural steering by c , naive/GLP-corrected, n=100 per cell. Structural column: DSSP secondary-structure fraction (per-chain residue fraction in that class) for Helix and Beta strand; EMBOSS pepcoil hit rate for Coiled coil; and, for Zinc finger, the fraction of sequences with any PF00096 zf-C2H2 profile-HMM alignment (a graded measure, no complete motif forms).
Figure 9: 1-D probing mean test AUC by method and host (ESM-2-8M solid, ESM-2-650M hatched; 95% bootstrap CI). GLP meta-neurons lead at both host sizes.
ESM-2-8M
ESM-2-650M
Concept
GLP
SAE
FFN
Res.
GLP
SAE
FFN
Res.
Domain / motif (n=92)
0.904 *
0.843
0.879
0.833
0.940 *
0.879
0.867
0.895
Compositional bias_Acidic residues
0.974
0.972
0.920
0.920
0.983
0.959
0.942
0.888
Compositional bias_Basic and acidic residues
0.818
0.718
0.745
0.783
0.748
0.664
0.682
0.680
Compositional bias_Basic residues
0.955
0.935
0.918
0.807
0.933
0.795
0.786
0.830
Compositional bias_Gly residues
0.939
0.890
0.891
0.916
0.937
0.859
0.829
0.835
Appendix
Table 8: Per-concept test AUC, all 110 concepts, both hosts. Bold marks the highest value in each row within a host. Shaded rows are category means; * marks a lead over the runner-up with p<0.05 (two-sided paired Wilcoxon over the category’s concepts, uncorrected). Categories with n≤4 cannot reach this threshold even if the leader wins every concept.
ESM-2-8M
ESM-2-650M
Property
GLP
SAE
FFN
Res.
GLP
SAE
FFN
Res.
Aliphatic index
0.697
0.601
0.585
0.651
0.593
0.691
0.403
0.486
Aromaticity
0.765
0.717
0.522
0.465
0.631
0.717
0.339
0.420
GRAVY (hydropathy)
0.803
0.638
0.648
0.570
0.684
0.562
0.465
0.401
Helix fraction
0.594
0.563
0.500
0.341
0.425
0.426
0.339
0.316
Instability index
0.575
0.578
0.567
0.482
0.524
0.524
0.450
0.474
Appendix
Table 9: Per-property test Spearman ρ , all 10 continuous properties, both hosts. The shaded row is the mean, with the Wilcoxon p for the leader against the runner-up.
ESM-2-8M
ESM-2-650M
Concepts tested
104 of 110
105 of 110
R2 , median [IQR]
0.756 [0.706, 0.842]
0.725 [0.627, 0.799]
Concepts at k∗=2/5/10/20
7/23/36/28
14/43/32/12
No k≤20 reaches it
10
4
short by, median
0.0035
0.0041
largest gap
Active site_Proton acceptor (0.171)
Domain_RRM 1 (0.054)
Appendix
Table 10: Falsification tests on the concepts GLP wins. R2 : meta-neuron regressed on the full SAE dictionary (held-out). k∗ : fewest top- k SAE features whose joint probe matches its test AUC (“none” if no k≤20 does).