Mutual information (MI) estimation is used to probe what neural representations encode, including in protein language models, but variational MI estimators have variance that scales at least exponentially in the true MI. We audit a prior neural-MI study of order-resolved epistasis in ESM-2 on the four-site GB1 landscape. Run-to-run differences show no systematic bias and a magnitude comparable to ordinary cross-seed retraining variability, and the evaluation-pairing scheme shifts scores substantially. At the original dimensionality, two of three alternative estimator families stabilize the estimate across seeds and one does not. In a synthetic calibration at the representation dimensionalities of two other protein language models (a Gaussian channel with known MI; no model representations), all three families recover materially less signal than required, despite correct signal ordering and stable seeds. A control critic given the true signal direction, which also reduces its input to one dimension, recovers 93.3% and 98.7% of the analytic MI at gain 1.5 (at 5,120 and 4,096 dimensions), so the signal is recoverable with privileged information, but the control does not identify why blind estimation fails. We package the checks into a reusable diagnostic protocol and conclude that even a stabilized estimator is not currently sufficient for order-resolved dependence claims made under blind search at this dimensionality and sample size.
Figures & tables
Figure 1: Measurement framework. Fixed positional subsets connect experimental fitness effects to frozen model representations via a shared Walsh transform, fit separately per layer and subset to produce a held-out dependence estimate Dℓ,S . Two distinct statistics are computed from the same Dℓ,S grid – peak-to-final retention and direct score contrast at a fixed layer – answering different questions rather than being sign-flipped versions of one statistic; both require synthetic calibration matched to the specific statistic and sample regime before any biological interpretation.
Figure 2: GB1 and its binding partner, the Fc fragment of IgG. GB1 is a 56-residue domain comprising a four-stranded β -sheet packed against a single α -helix; the four positions mutated throughout this protocol (39, 40, 41, 54) lie at the binding interface with Fc.
Figure 3: Order-resolved Walsh decomposition of a four-site GB1 cube. Each WT-anchored cube selects one alternative amino acid at each site and contains all 16 WT/alternative combinations. The 15 nonconstant components comprise four first-order, six second-order, four third-order, and one fourth-order positional subset. Filled circles indicate membership in the Walsh basis subset, not mutation states. Every coefficient uses all 16 vertices. In the diagram, c indexes cubes, b denotes a binary genotype, fc(b) denotes its fitness, and S denotes a positional subset. The fitness coefficient εc,S corresponds to EC,S in the main text. The same transform is applied coordinate-wise to the representations.
Pairing scheme
Old statistic COV
New (pooled) COV
New (equal-weighted) COV
Cyclic
0.540
0.272
0.161
Derandomized
14.157
0.152
0.060
Full permutation
3.240
0.342
0.252
Table 1: Retraining stability by pairing scheme (Experiment 1). Coefficient of variation (COV) of ΔHO−LO across the 3 critic-retraining seeds, for the old (peak-ratio) statistic and the new (direct-comparison) statistic under both weighting conventions. Lower COV indicates greater stability across independent retrains of the same critic.
k
Main run
replication
1
0.0060
0.0051
2
0.0148
0.0138
3
0.0101
0.0141
4
0.0238
0.0140
Table 2: Training variability: mean cross-seed variance per (layer, subset) cell, by interaction order k , computed separately for each run. Both runs use the standard cyclic-shift evaluation.
k
n matched
Mean diff.
Mean ∣ diff. ∣
SD of diff.
1
372
0.322
0.322
0.091
2
558
0.333
0.334
0.148
3
372
0.346
0.346
0.155
4
93
0.301
0.301
0.124
Table 3: Pairing sensitivity: matched differences (derandomized − cyclic) on the replication run, by interaction order k .
k
n matched
Mean diff.
Mean ∣ diff. ∣
SD of diff.
1
372
-0.003
0.079
0.104
2
558
-0.005
0.081
0.121
3
372
-0.013
0.082
0.127
4
93
-0.006
0.090
0.134
Table 4: Run reproducibility: matched differences (replication − main), cyclic scheme, by interaction order k .
Estimator
Max SD
ΔHO−LO
Strong min.
Null max.
InfoNCE
0.0305
−0.9378
0.5319
0.0014
Bounded-DV
0.0087
−0.9705
0.5278
0.0022
SMILE
0.0070
−0.9641
0.5264
0.0023
Table 5: ORBIT-Synth validation results, all three alternative estimators (Experiment 2). Columns: maximum cross-seed standard deviation; the synthetic ΔHO−LO selectivity contrast; the minimum recovered estimate on the strong synthetic channel; the maximum ∣ estimate ∣ on the null-control channel. All four acceptance gates pass for each estimator; full gate-by-gate detail (Walsh reconstruction, bottleneck localization, order-level signal-minus-null) is in each estimator’s archived validation manifest.
Estimator
Order 1
Order 2
Order 3
Order 4
DV/MINE
0.9953
0.8820
0.7284
0.6622
Bounded-DV
0.9460
0.8845
0.7552
0.6749
SMILE
1.2097
1.0792
0.9449
0.8233
Table 6: Mean held-out MI estimate, in nats, by interaction order at layer 30, all three estimators, extracted directly from the same validated grid Figure 7 and Figure 8 (Appendix D ) are plotted from, at the specific layer Table 10 also uses. Each cell averages seeds within subset first, then subsets within order (12 seed-subset cells for orders 1 and 3, 18 for order 2, 3 for order 4). No threshold filtering is applied.
k
ESM-1v layer 0
ESM-1v layers 1–33 (SD)
ProtT5 layer 0
ProtT5 layers 1–24 (SD)
1
0.4773
0.9911 (0.0017)
0.4774
0.9834 (0.0013)
2
-0.0015
0.9738 (0.0023)
-0.0015
0.9608 (0.0040)
3
-0.0009
0.9339 (0.0057)
-0.0009
0.9186 (0.0057)
4
-0.0000
0.8531 (0.0154)
-0.0000
0.8464 (0.0140)
LO (1,2) mean
–
0.9824
–
0.9721
HO (3,4) mean
–
0.8935
–
0.8825
Table 7: Ridge mean held-out R2 ( Dℓ,S ) by interaction order, ESM-1v and ProtT5. Layer 0 is reported separately, matching the paper’s own ESM-2 convention; the main summary covers layers 1–33 (ESM-1v) / 1–24 (ProtT5). Mean and SD for the layers-1- N rows are computed across each layer’s own per-order mean (layer-to-layer variation), not across individual layer–subset cells. The LO/HO contrast weights each interaction order equally (mean of the two order-level means), matching the paper’s existing ESM-2 convention, not a pooled mean over all cells.
Estimator
n (cyc/derand, cyc/fullperm)
Cyclic / derandomized
Cyclic / full-permutation
DV/MINE
1350, 1350
69.7% (53.5%–83.1%)
70.1% (51.1%–87.9%)
Bounded-DV
1350, 1350
71.7% (58.2%–83.2%)
71.8% (57.4%–85.5%)
SMILE
1350, 1350
73.8% (62.7%–83.9%)
74.3% (62.3%–88.2%)
Table 8: Cyclic-shift pairing sensitivity, by estimator. Ratio of the standard cyclic-shift test_mi to the same critic’s estimate under two alternative negative-pairing schemes. Reported statistic is the MEAN of the per-fit ratio (5th–95th percentile range shown in parentheses); n is fits actually used per comparison after excluding layer 0 and any non-finite or near-zero-denominator cases.
Order k
Layer 0
Mean, layers 1–30
SD, layers 1–30
1
0.4773
0.9833
0.0032
2
−0.0015
0.9619
0.0037
3
−0.0009
0.9140
0.0065
4
−0.0000
0.8239
0.0184
Table 9: Experiment 5 (Ridge regression robustness check). Dℓ,S is held-out R2 (linear decodability), a different statistical quantity from the MI estimates DV/MINE and its alternatives report in nats. Layer 0 column: mean Dℓ,S at the raw embedding layer. Layers 1–30 columns: mean and SD across the 30 per-layer means, subsets averaged within each layer first.
Figure 4: Linear decodability by layer and interaction order, layers 1–30. Layer 0 is a floor case (near zero for orders 2–4) reported separately in Table 9 . Shading: ± 1 sample SD across subsets at each layer (descriptive, not a confidence interval).
Estimator
Seed 1 (11)
Seed 2 (23)
Seed 3 (37)
Cross-run
DV/MINE (baseline)
0.209
0.143
0.284
Stable (3/3)
InfoNCE
−0.116
0.041
0.091
Sign-agree (3/3) †
Bounded-DV
0.204
0.134
0.172
Stable (3/3)
SMILE
0.225
0.235
0.172
Stable (3/3)
Table 10: ΔHO−LO stability across estimators (Experiment 2), ESM-2 only. Values are the pooled raw-score mean difference (lower-order minus higher-order), evaluated at layer 30, from Section 3.2 ’s direct-comparison diagnostic, not the original ratio-based statistic; the sign convention is therefore reversed relative to ΔHO−LO itself, so a positive value here indicates lower-order exceeding higher-order, the same direction as the original finding. DV/MINE row uses the cyclic evaluation-pairing scheme, matching the single convention the other three estimators’ main grids were run under. Cross-run entries report sign agreement (matched seed pairs) between each estimator’s original grid and an independent second run; for Bounded-DV and SMILE this reuses their pairing-sensitivity rerun (Table 8 ) rather than a dedicated rerun.
Figure 5: Strong-signal-margin gate: every replicate/gain comparison for both models, across DV/MINE, Bounded-DV, SMILE, and the privileged Oracle diagnostic. Each point represents a replicate-level estimate, averaged across critic seeds, minus that replicate’s corresponding null estimate. The run passes this margin gate only if all six comparisons meet the 0.05 requirement – a single point below the dashed line fails the gate for that estimator even if the other five pass. Oracle receives a privileged 1-D projection onto the known true signal direction, not the full-dimensional blind input the other three estimators receive – it is a privileged diagnostic, not a fourth blind attempt. Margins were recomputed from saved per-replicate summaries; each run’s overall margin-gate outcome agrees with its manifest. Shaded vertical bands group each estimator’s six points – point shape and color encode gain and pass/fail only, not estimator identity, so the band is what shows which estimator a point belongs to.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Estimator
Rep.
Gain
Estimate
Null est.
Margin
Shortfall
Comparison passes
ProtT5
DV/MINE
101
1.0
0.039377
-0.000219
0.039596
0.010404
No
ProtT5
DV/MINE
101
1.5
0.142699
-0.000219
0.142918
-0.092918
Yes
ProtT5
DV/MINE
211
1.0
0.047836
-0.000170
0.048006
0.001994
No
ProtT5
DV/MINE
211
1.5
0.149701
-0.000170
0.149871
-0.099871
Yes
ProtT5
DV/MINE
307
1.0
0.040352
-0.000121
0.040474
0.009526
No
ProtT5
DV/MINE
307
1.5
0.127060
-0.000121
0.127181
-0.077181
Yes
Appendix
Table 11: Full strong-signal-margin gate data: every replicate/gain comparison underlying Figure 5 , independently recomputed from the real per-replicate summaries ( replicate_summary.csv ) on both models, for all four conditions. Margin = estimate − that replicate’s own null estimate; shortfall = 0.05 − margin. Pass/fail uses the unrounded comparison estimate ≥ null estimate + 0.05; displayed values are rounded to six decimal places. Oracle is a privileged diagnostic – given the true signal direction rather than finding it blind – not a fourth blind estimator. A run passes this margin gate only when all six of its comparisons pass; manifest agreement here concerns that gate specifically, not overall calibration.
Figure 6: Evaluation-pairing schemes, holding the trained critic fixed. Rows index held-out cubes under a fixed layer and subset. The joint term Ri↔Ei is unchanged across every scheme. Product-term pairings only: original cyclic shifts (one illustrative shift s=1 shown; the configured shifts are evaluated and summarized); joint reorder + cyclic shifts (the same seeded permutation applied to R and E , then E is rolled – true pairs are preserved under the permutation, not the roll); full random permutations (independent seeded draws of E only, scored and summarized – fixed points may occur, as shown for R2 → E2).
Figure 7: Order-resolved dependence by layer on ESM-2, for DV/MINE, Bounded-DV, and SMILE. For each (layer, order) cell, critic seeds are averaged within each subset first, then the mean and standard deviation are taken across subsets; shading shows ±1 sample SD across subsets after this seed-averaging step and is descriptive, not a confidence interval. Order 4 has only one positional subset, so no band can be computed and none is drawn – this reflects the absence of subset-level spread to measure, not that order 4’s estimate is more certain than the others. The bands shown describe subset-to-subset heterogeneity only; they do not display critic-seed variability, which is characterized separately in Section 3.2 . These curves summarize all scores without threshold filtering and show raw score trajectories by layer, not the fraction of subsets meeting the eligibility threshold used elsewhere (Figure 1 ). Lower-order mean scores generally sit above higher-order scores across depth, though orders 3 and 4 sometimes cross; the pattern is not a strict separation holding at every individual layer.
Figure 8: Cross-estimator comparison by interaction order on ESM-2. For each (layer, order) cell and each estimator, critic seeds are averaged within each positional subset first, then subsets are averaged within the order; the resulting mean trajectories for DV/MINE, Bounded-DV, and SMILE are plotted directly against each other within each order panel. No threshold filtering is applied and no uncertainty bands are shown – this figure displays mean score trajectories only, to make cross-estimator agreement or divergence easy to read at a glance; neither subset-to-subset nor critic-seed variability is depicted here (see Figure 7 and Section 3.2 respectively). SMILE generally reports higher estimates in these runs. Because the estimator procedures differ and ground-truth MI is unavailable for these biological data, higher scores do not establish greater estimation accuracy.
Mutual information estimation measures statistical dependence between variables from data, supporting analysis, prediction, and information-aware learning.