A low-rank decoder bottleneck bounds reliability in foundation-model perturbation prediction
Organizations: Computer Science and Engineering Division, University of Michigan, Ann Arbor, MI 48109 · Gilbert S. Omenn Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, MI 48109
Abstract
Foundation-model predictors of perturbation response are evaluated primarily by aggregate accuracy. Using their predictions to prioritize experiments requires assessing each prediction's reliability. We audit the state-transition (ST) decoder used in STATE across three encoders (SE-600M, scGPT, and Tahoe-x1) and four cell lines under CRISPRi. The decoder's first attention layer has a low-rank output. Projecting this output onto 4-32 fixed directions estimated from training activations preserves mean per-perturbation Pearson delta correlation (PDC). This low-rank attention bottleneck bounds the decoder's reliability by distance to training responses: PDC declines as the cosine distance from a perturbation's observed response to its nearest training response increases. We propose predicted-NN-distance, the cosine distance from a predicted response to its nearest training response, as a per-prediction confidence score. The score has a mean absolute Spearman correlation of 0.81 with PDC, close to the 0.88 obtained with the observed response. Thresholds calibrated on this score transfer across encoders and cell lines, and the score remains predictive of PDC for drug perturbations.
Figures & tables
| held-out perturbations | far | ||||||
|---|---|---|---|---|---|---|---|
| HepG2 | Jurkat | RPE1 | K562 | mean | mean | ||
| Train-mean response | 0.293 | 0.171 | 0.416 | 0.148 | 0.257 | 0.073 | |
| CPA [ 30 ] | 0.403 | 0.330 | 0.501 | 0.302 | 0.384 | 0.117 | |
| SE ST | full | 0.388 | 0.306 | 0.475 | 0.327 | 0.374 | 0.093 |
| rank-1 | 0.385 | 0.298 | 0.474 | 0.331 | 0.372 | 0.092 | |
| ridge | 0.369 | 0.334 | 0.473 | 0.331 | 0.377 | 0.148 | |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Encoders (frozen) | |
| SE-600M output dimension | |
| scGPT output dimension | |
| Tahoe-x1 ( tx-70m ) output dimension | |
| Inference | |
| Control cells per perturbation ( ) | |
| Probe | Settings |
|---|---|
| MLP (main) | 2 layers, hidden , ReLU, dropout , Adam, learning rate , batch , epochs |
| Ridge | closed form, penalty chosen by -fold grouped cross-validation on PDC |
| Large MLP | 3 layers, hidden ( – M parameters), GELU, dropout , AdamW with cosine schedule, epochs, early stopping on of training rows (patience ) |
| Interaction | linear term plus , rank , : 2-layer MLPs (hidden , GELU, dropout ), AdamW (learning rate , weight decay ), epochs, early stopping as above |
| All held-out | Far ( cosine) | |||||||||
| K562 | RPE1 | HepG2 | Jurkat | Mean | K562 | RPE1 | HepG2 | Jurkat | Mean | |
| Perturbations ( ) | 963 | 1,029 | 903 | 1,077 | — | 521 | 274 | 403 | 549 | — |
| Shuffled-perturbation lookup | 0.072 | 0.240 | 0.151 | 0.090 | 0.138 | 0.032 | 0.034 | 0.040 | 0.044 | 0.037 |
| Train-mean response | 0.148 | 0.416 | 0.293 | 0.171 | 0.257 | 0.065 | 0.072 | 0.078 | 0.078 | 0.073 |
| Co-expression (gene–gene) | 0.153 | 0.419 | 0.292 | 0.169 | 0.258 | 0.067 | 0.073 | 0.077 | 0.076 | 0.073 |
| CPA [ 30 ] | 0.302 | 0.501 | 0.403 | 0.330 | 0.384 | 0.117 | 0.086 | 0.117 | 0.147 | 0.117 |
| Cell | ||||||||
|---|---|---|---|---|---|---|---|---|
| HepG2 | 945 | 395 | ||||||
| Jurkat | 1084 | 453 | ||||||
| RPE1 | 1061 | 438 | ||||||
| K562 | 968 | 415 |
| Top direction | Random direction | ||||||||||||||||
| Encoder | Cell line | L0 | L1 | L2 | L3 | L4 | L5 | L6 | L7 | L0 | L1 | L2 | L3 | L4 | L5 | L6 | L7 |
| SE+ST | HepG2 | 99 | 98 | 97 | 98 | 97 | 99 | 99 | 99 | 4 | 99 | 100 | 101 | 97 | 99 | 95 | 98 |
| SE+ST | Jurkat | 96 | 98 | 96 | 99 | 98 | 97 | 97 | 98 | -3 | 95 | 94 | 96 | 96 | 94 | 96 | 97 |
| SE+ST | RPE1 | 99 | 99 | 99 | 98 | 99 | 101 | 99 | 100 | 2 | 103 | 99 | 100 | 99 | 99 | 99 | 98 |
| SE+ST | K562 | 102 | 99 | 98 | 100 | 99 | 99 | 99 | 99 | -4 | 98 | 98 | 98 | 99 | 99 | 100 | 100 |
| Tahoe+ST | HepG2 | 97 | 99 | 95 | 93 | 89 | 87 | 95 | 100 | 42 | 96 | 88 | 94 | 91 | 94 | 97 | 99 |
| Encoder | Donor | HepG2 | Jurkat | RPE1 | K562 |
|---|---|---|---|---|---|
| SE-600M | other cell lines | ||||
| same cell line | |||||
| Tahoe-x1 | other cell lines | ||||
| same cell line | |||||
| scGPT | other cell lines | ||||
| same cell line |
| Cell line | rank | rank | rank | resampled | Tahoe ST | ||
|---|---|---|---|---|---|---|---|
| HepG2 | 4.29 | 8.26 | 0.448 | 0.548 | 0.661 | 0.854 | 0.446 |
| Jurkat | 10.96 | 19.85 | 0.337 | 0.496 | 0.624 | 0.840 | 0.396 |
| RPE1 | 2.60 | 5.37 | 0.566 | 0.631 | 0.736 | 0.923 | 0.558 |
| K562 | 13.05 | 19.81 | 0.290 | 0.457 | 0.586 | 0.830 | 0.393 |
| Encoder | HepG2 | Jurkat | RPE1 | K562 |
|---|---|---|---|---|
| SE-600M | ||||
| Tahoe-x1 | ||||
| scGPT |
| Pert | PDC | test cells | Jaccard top-20 | cos( , ) | shared sign |
|---|---|---|---|---|---|
| RPL6 | 0.980 | 106 | 0.67 | 0.962 | 16 |
| RPL23A | 0.975 | 67 | 0.67 | 0.952 | 16 |
| RPL35A | 0.973 | 174 | 0.54 | 0.943 | 14 |
| RPL23 | 0.973 | 134 | 0.60 | 0.943 | 15 |
| RPL7 | 0.971 | 110 | 0.60 | 0.933 | 15 |
| RRS1 | 0.970 | 132 | 0.67 | 0.948 | 16 |
| gene | loading | gene | loading | gene | loading | gene | loading |
|---|---|---|---|---|---|---|---|
| TUBA1B | STMN1 | H2AFZ | MALAT1 | ||||
| HMGB2 | TUBB | HMGB1 | CENPF | ||||
| NEAT1 | HIST1H4C | PCLAF | MDM2 | ||||
| MKI67 | HMGN2 | BIRC5 | NASP | ||||
| CALR | MCM3 | PTMA | HSP90B1 | ||||
| SCD | ACTG1 | TOP2A | TPX2 |
| Cell line | Top-FDR Hallmark category | |
|---|---|---|
| HepG2 | E2F-Targets | |
| RPE1 | E2F-Targets | |
| Jurkat | Myc-Targets-V1 | |
| K562 | mTORC1-Signaling | |
| Hypoxia (tied) |
| percentile | cosine | Jaccard top-20 | shared+sign / 20 |
|---|---|---|---|
| PDC decile | PDC med | cosine | cosine median | Jaccard med |
|---|---|---|---|---|
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 | ||||
| 6 |
| Encoder | Cell line | (a) | (b) | (c) | (d) | (e) |
|---|---|---|---|---|---|---|
| SE | HepG2 | 0.844 | 0.769 | 0.827 | 0.322 | 0.744 |
| SE | Jurkat | 0.832 | 0.692 | 0.819 | 0.498 | 0.652 |
| SE | RPE1 | 0.817 | 0.822 | 0.797 | 0.707 | 0.712 |
| SE | K562 | 0.878 | 0.751 | 0.870 | 0.511 | 0.719 |
| Tahoe | HepG2 | 0.840 | 0.823 | 0.808 | 0.585 | 0.702 |
| Tahoe | Jurkat | 0.850 | 0.828 | 0.822 | 0.688 | 0.662 |
| Encoder | Cell line | unbin. | Q1 | Q2 | Q3 | Q4 | Q5 | MOD | STR | |
|---|---|---|---|---|---|---|---|---|---|---|
| SE | HepG2 | 945 | ✓ | ✓ | ||||||
| SE | Jurkat | 1084 | ✓ | ✓ | ||||||
| SE | RPE1 | 1061 | ✓ | — | ||||||
| SE | K562 | 968 | ✓ | ✓ | ||||||
| Tahoe-x1 | HepG2 | 945 | ✓ | — | ||||||
| Tahoe-x1 | Jurkat | 1084 | ✓ | ✓ |
| Bin | Cross-config median | IQR ( ) | ||
|---|---|---|---|---|
| Q1 | ||||
| Q2 | ||||
| Q3 | ||||
| Q4 | ||||
| Q5 |
| Cell line | target | mean stratified | mean unbinned | attenuation ratio |
|---|---|---|---|---|
| HepG2 | ||||
| Jurkat | ||||
| RPE1 | ||||
| K562 |
| Learn-then-Test | Decile rule | ||||
|---|---|---|---|---|---|
| Encoder | Cell line | Accepted | PDC | Accepted | PDC |
| SE | HepG2 | 0.696 | 0.520 | 0.465 | 0.654 |
| SE | Jurkat | 0.380 | 0.574 | 0.342 | 0.600 |
| SE | K562 | 0.506 | 0.553 | 0.419 | 0.603 |
| SE | RPE1 | 0.987 | 0.475 | 0.612 | 0.651 |
| Tahoe | HepG2 | 0.897 | 0.494 | 0.584 | 0.646 |
| Exphormer | GAT | |
| Mean PDC | 0.600 | 0.595 |
| Oracle distance vs PDC, | 0.821 | 0.802 |
| Predicted vs observed distance, | 0.607 | 0.443 |
| Predicted-NN-distance vs PDC, | 0.697 | 0.481 |
| Top- singular direction | 0.366 | 0.344 |
| Random direction, rank | 0.031 |
| Cell line | interval | PR observed | PR predicted | |
|---|---|---|---|---|
| C32 | 3.48 | 2.39 | ||
| HOP62 | 5.55 | 4.44 | ||
| HepG2-C3A | 5.60 | 4.12 | ||
| Hs 766T | 7.79 | 5.92 | ||
| PANC-1 | 7.03 | 5.37 |
| Context size | ||||||||
|---|---|---|---|---|---|---|---|---|
| Encoder | Cell line | Pseudobulk | Single cell | 16 | 32 | 64 | 128 | 256 |
| SE | HepG2 | 0.844 | 0.744 | 0.852 | 0.841 | 0.850 | 0.837 | 0.841 |
| SE | Jurkat | 0.832 | 0.647 | 0.862 | 0.852 | 0.849 | 0.855 | 0.858 |
| SE | K562 | 0.878 | 0.753 | 0.878 | 0.884 | 0.879 | 0.890 | 0.881 |
| SE | RPE1 | 0.817 | 0.765 | 0.811 | 0.823 | 0.817 | 0.822 | 0.813 |
| Tahoe | HepG2 | 0.840 | 0.712 | 0.870 | 0.855 | 0.841 | 0.833 | 0.839 |
| SE ST | Tahoe ST | scGPT ST | ||||
|---|---|---|---|---|---|---|
| Cell line ( ) | PDC | PDC | PDC | |||
| HepG2 ( ) | 0.251 | 0.316 | 0.316 | |||
| Jurkat ( ) | 0.279 | 0.281 | 0.170 | |||
| K562 ( ) | 0.114 | 0.176 | 0.133 | |||
| RPE1 ( ) | 0.471 | 0.463 | 0.466 | |||
Single-Cell Perturbation Modeling
Single-cell perturbation modeling predicts how genetic or chemical interventions change cellular states across cells and contexts.
See all 70 papers