When Can You Trust Neural Critics? Auditing Mutual Information Estimates of Higher-Order Epistasis in Protein Language Models
Organizations: University of Central Florida
Abstract
Mutual information (MI) estimation is used to probe what neural representations encode, including in protein language models, but variational MI estimators have variance that scales at least exponentially in the true MI. We audit a prior neural-MI study of order-resolved epistasis in ESM-2 on the four-site GB1 landscape. Run-to-run differences show no systematic bias and a magnitude comparable to ordinary cross-seed retraining variability, and the evaluation-pairing scheme shifts scores substantially. At the original dimensionality, two of three alternative estimator families stabilize the estimate across seeds and one does not. In a synthetic calibration at the representation dimensionalities of two other protein language models (a Gaussian channel with known MI; no model representations), all three families recover materially less signal than required, despite correct signal ordering and stable seeds. A control critic given the true signal direction, which also reduces its input to one dimension, recovers 93.3% and 98.7% of the analytic MI at gain 1.5 (at 5,120 and 4,096 dimensions), so the signal is recoverable with privileged information, but the control does not identify why blind estimation fails. We package the checks into a reusable diagnostic protocol and conclude that even a stabilized estimator is not currently sufficient for order-resolved dependence claims made under blind search at this dimensionality and sample size.
Figures & tables
| Pairing scheme | Old statistic COV | New (pooled) COV | New (equal-weighted) COV |
|---|---|---|---|
| Cyclic | |||
| Derandomized | |||
| Full permutation |
| Main run | replication | |
|---|---|---|
| 1 | 0.0060 | 0.0051 |
| 2 | 0.0148 | 0.0138 |
| 3 | 0.0101 | 0.0141 |
| 4 | 0.0238 | 0.0140 |
| matched | Mean diff. | Mean diff. | SD of diff. | |
|---|---|---|---|---|
| 1 | 372 | 0.322 | 0.322 | 0.091 |
| 2 | 558 | 0.333 | 0.334 | 0.148 |
| 3 | 372 | 0.346 | 0.346 | 0.155 |
| 4 | 93 | 0.301 | 0.301 | 0.124 |
| matched | Mean diff. | Mean diff. | SD of diff. | |
|---|---|---|---|---|
| 1 | 372 | -0.003 | 0.079 | 0.104 |
| 2 | 558 | -0.005 | 0.081 | 0.121 |
| 3 | 372 | -0.013 | 0.082 | 0.127 |
| 4 | 93 | -0.006 | 0.090 | 0.134 |
| Estimator | Max SD | Strong min. | Null max. | |
|---|---|---|---|---|
| InfoNCE | ||||
| Bounded-DV | ||||
| SMILE |
| Estimator | Order 1 | Order 2 | Order 3 | Order 4 |
|---|---|---|---|---|
| DV/MINE | 0.9953 | 0.8820 | 0.7284 | 0.6622 |
| Bounded-DV | 0.9460 | 0.8845 | 0.7552 | 0.6749 |
| SMILE | 1.2097 | 1.0792 | 0.9449 | 0.8233 |
| ESM-1v layer 0 | ESM-1v layers 1–33 (SD) | ProtT5 layer 0 | ProtT5 layers 1–24 (SD) | |
| 1 | 0.4773 | 0.9911 (0.0017) | 0.4774 | 0.9834 (0.0013) |
| 2 | -0.0015 | 0.9738 (0.0023) | -0.0015 | 0.9608 (0.0040) |
| 3 | -0.0009 | 0.9339 (0.0057) | -0.0009 | 0.9186 (0.0057) |
| 4 | -0.0000 | 0.8531 (0.0154) | -0.0000 | 0.8464 (0.0140) |
| LO (1,2) mean | – | 0.9824 | – | 0.9721 |
| HO (3,4) mean | – | 0.8935 | – | 0.8825 |
| Estimator | (cyc/derand, cyc/fullperm) | Cyclic / derandomized | Cyclic / full-permutation |
|---|---|---|---|
| DV/MINE | 1350, 1350 | 69.7% (53.5%–83.1%) | 70.1% (51.1%–87.9%) |
| Bounded-DV | 1350, 1350 | 71.7% (58.2%–83.2%) | 71.8% (57.4%–85.5%) |
| SMILE | 1350, 1350 | 73.8% (62.7%–83.9%) | 74.3% (62.3%–88.2%) |
| Order | Layer 0 | Mean, layers 1–30 | SD, layers 1–30 |
|---|---|---|---|
| 1 | |||
| 2 | |||
| 3 | |||
| 4 |
| Estimator | Seed 1 (11) | Seed 2 (23) | Seed 3 (37) | Cross-run |
|---|---|---|---|---|
| DV/MINE (baseline) | Stable (3/3) | |||
| InfoNCE | Sign-agree (3/3) † | |||
| Bounded-DV | Stable (3/3) | |||
| SMILE | Stable (3/3) |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Estimator | Rep. | Gain | Estimate | Null est. | Margin | Shortfall | Comparison passes |
|---|---|---|---|---|---|---|---|---|
| ProtT5 | DV/MINE | 101 | 1.0 | 0.039377 | -0.000219 | 0.039596 | 0.010404 | No |
| ProtT5 | DV/MINE | 101 | 1.5 | 0.142699 | -0.000219 | 0.142918 | -0.092918 | Yes |
| ProtT5 | DV/MINE | 211 | 1.0 | 0.047836 | -0.000170 | 0.048006 | 0.001994 | No |
| ProtT5 | DV/MINE | 211 | 1.5 | 0.149701 | -0.000170 | 0.149871 | -0.099871 | Yes |
| ProtT5 | DV/MINE | 307 | 1.0 | 0.040352 | -0.000121 | 0.040474 | 0.009526 | No |
| ProtT5 | DV/MINE | 307 | 1.5 | 0.127060 | -0.000121 | 0.127181 | -0.077181 | Yes |
Mutual Information Estimation
Mutual information estimation measures statistical dependence between variables from data, supporting analysis, prediction, and information-aware learning.
See all 23 papers