A low-rank decoder bottleneck bounds reliability in foundation-model perturbation prediction
Authors: Zheyu Zhang, Yuanhao Huang, Fan Feng, Jie Liu
Organizations: Computer Science and Engineering Division, University of Michigan, Ann Arbor, MI 48109 · Gilbert S. Omenn Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, MI 48109
Foundation-model predictors of perturbation response are evaluated primarily by aggregate accuracy. Using their predictions to prioritize experiments requires assessing each prediction's reliability. We audit the state-transition (ST) decoder used in STATE across three encoders (SE-600M, scGPT, and Tahoe-x1) and four cell lines under CRISPRi. The decoder's first attention layer has a low-rank output. Projecting this output onto 4-32 fixed directions estimated from training activations preserves mean per-perturbation Pearson delta correlation (PDC). This low-rank attention bottleneck bounds the decoder's reliability by distance to training responses: PDC declines as the cosine distance from a perturbation's observed response to its nearest training response increases. We propose predicted-NN-distance, the cosine distance from a predicted response to its nearest training response, as a per-prediction confidence score. The score has a mean absolute Spearman correlation of 0.81 with PDC, close to the 0.88 obtained with the observed response. Thresholds calibrated on this score transfer across encoders and cell lines, and the score remains predictive of PDC for drug perturbations.
Figures & tables
Figure 1: Per-prediction reliability of encoder + ST models decays smoothly with gene-space distance to the training perturbations. (A) Encoder coupled with ST decoder for perturbation prediction. An upstream encoder (any single-cell foundation model or cell embedding, e.g., SE-600M, Tahoe-x1, scGPT) embeds each control cell, and the perturbation embedding is added to every cell embedding ( ⊕ ). The ST decoder ( 8 attention blocks) and a gene decoder map these to predicted cells. Predictions are averaged into a pseudobulk for evaluation (§ 3 ). Each cell is drawn as a vector. Before L0 (black outline) the cell vectors point in different directions, and after it they point along one shared direction with different lengths. This is the low-rank output of the L0 self-attention (§ 4.2 ). (B) HepG2. Each gray dot is one of the 945 held-out perturbations under Tahoe-x1 + ST, plotted as Pearson delta correlation (PDC, Eq. ( 1 )) against the cosine distance in gene space from its response to the nearest training response. Lines are kernel-smoothed trends per encoder (Spearman ρ in the legend). RPL6 (small distance) is predicted reliably and SNAPC5 (large distance) is not. The pattern replicates across all four cell lines (§ B.2 ). The deployable analog (Definition 1 ) and the calibrated threshold are developed in § 4.3 .
held-out perturbations
far
HepG2
Jurkat
RPE1
K562
mean
mean
Train-mean response
0.293
0.171
0.416
0.148
0.257
0.073
CPA [ 30 ]
0.403
0.330
0.501
0.302
0.384
0.117
SE + ST
full
0.388
0.306
0.475
0.327
0.374
0.093
rank-1
0.385
0.298
0.474
0.331
0.372
0.092
ridge
0.369
0.334
0.473
0.331
0.377
0.148
Table 1: Comparison with simple predictors on held-out perturbations. Mean PDC (Eq. ( 1 )) over held-out perturbations and over those far from every training response (cosine distance above 0.5 ). The decoders score in the range of the simple predictors, and every predictor loses most of its accuracy far from training. Per-cell-line far values, counts, and further baselines are in Appendix B.1 .
Figure 2: How the decoder reads the perturbation. (A) PDC after replacing the output of one attention layer with a single random direction, relative to the decoder without the intervention (mean over the four cell lines; band, range). (B) Per-perturbation PDC with and without an intervention on the top L0 component, which keeps its direction and sets its size to the mean over perturbations (left), or keeps its size and takes the direction of another perturbation (right). Numbers give the mean change. (C) PDC after projecting the L0 output onto k directions shared by all perturbations and estimated from training activations, or onto k random directions, relative to the decoder without the intervention. For scGPT + ST one direction is too few even for each perturbation’s own top direction (Table 1 ). The perturbation enters at L0 as a direction in a small shared subspace, and the perturbation-specific signal lies in this direction.
Figure 3: Selecting predictions with predicted-NN-distance. (A) Mean PDC of the accepted predictions when held-out predictions are accepted in order of increasing distance, averaged over the four cell lines of each encoder; solid, predicted-NN-distance; dashed, the oracle distance from the observed response. At share 1 every curve reaches the mean PDC of all predictions. (B) Mean PDC of the accepted and rejected predictions under the pooled threshold τ≈0.10 in each encoder–cell configuration; numbers give the share accepted. The score selects predictions nearly as well as the oracle, and one threshold selects accurate predictions in every configuration.
Figure 4: Predicted-NN-distance outside the audited setting. Mean PDC in six equal-count bins of predicted-NN-distance. (A) Arc’s ST model on Tahoe-100M, drug–dose conditions in five test cell lines, one line per cell line. (B) PDC computed against single cells instead of the pseudobulk, one line per encoder, mean over cell lines. (C) ST decoders retrained with one gene family held out, scored on that family, one line per encoder, mean over cell lines. The score correlates with PDC in A and B (Spearman ∣ρ∣=0.57 – 0.85 ) and more weakly after family-held-out retraining (C; negative in 9 of 12 configurations).
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Encoders (frozen)
SE-600M output dimension
2,058
scGPT output dimension
512
Tahoe-x1 ( tx-70m ) output dimension
512
Inference
Control cells per perturbation ( S )
64
Appendix
Table 2: Audit configurations and hyperparameters. The encoder changes across configurations. The ST decoder architecture and training recipe are the same in every run.
closed form, penalty chosen by 5 -fold grouped cross-validation on PDC
Large MLP
3 layers, hidden 2,048 ( 17 – 23 M parameters), GELU, dropout 0.1 , AdamW with cosine schedule, 120 epochs, early stopping on 10% of training rows (patience 15 )
Interaction
linear term plus B⊤(ϕb(control)⊙ϕp(perturbation)) , rank r∈{0,4,16,64} , ϕ : 2-layer MLPs (hidden 128 , GELU, dropout 0.1 ), AdamW (learning rate 5×10−4 , weight decay 10−5 ), 120 epochs, early stopping as above
Appendix
Table 3: Probe settings.
All held-out
Far ( >0.5 cosine)
K562
RPE1
HepG2
Jurkat
Mean
K562
RPE1
HepG2
Jurkat
Mean
Perturbations ( n )
963
1,029
903
1,077
—
521
274
403
549
—
Shuffled-perturbation lookup
0.072
0.240
0.151
0.090
0.138
0.032
0.034
0.040
0.044
0.037
Train-mean response
0.148
0.416
0.293
0.171
0.257
0.065
0.072
0.078
0.078
0.073
Co-expression (gene–gene)
0.153
0.419
0.292
0.169
0.258
0.067
0.073
0.077
0.076
0.073
CPA [ 30 ]
0.302
0.501
0.403
0.330
0.384
0.117
0.086
0.117
0.147
0.117
Appendix
Table 4: Mean PDC on all held-out perturbations and on the far slice. Same gene panel and perturbation support for every row; scVI was scored against a separately built observed response. GEARS was scored on K562 only.
Figure 5: Reliability-distance relationship across all four cell lines. Each panel plots held-out per-perturbation PDC against cosine distance to the nearest training perturbation in 2,000 -HVG gene-expression space. Gray dots in the background are individual perturbations under Tahoe-x1 + ST. Lines are kernel-smoothed trends per encoder with 95% bootstrap bands, and the lower-left key of each panel gives the per-encoder Spearman ρ . RPL6 and SNAPC5 are annotated on the HepG2 panel as the running example (§ 1 ). Every panel shows the HepG2 pattern: PDC falls smoothly with distance and the three encoders agree. The weakest configuration is scGPT/K562, with ρ=−0.73 .
Cell
ntest
ntrain
ρPDC,dHVG
ρPDC,dTahoe
ρPDC,∣cos∣
rPDC,dHVG∣∣cos∣
rPDC,∣cos∣∣dHVG
rPDC,dTahoe∣∣cos∣
HepG2
945
395
−0.92
−0.88
0.86
−0.64
0.06
−0.38
Jurkat
1084
453
−0.90
−0.76
0.74
−0.76
0.09
−0.33
RPE1
1061
438
−0.91
−0.86
0.84
−0.63
0.14
−0.30
K562
968
415
−0.92
−0.81
0.78
−0.79
0.00
−0.45
Appendix
Table 5: Tahoe + ST: association of PDC with the distance to the nearest training response in gene space ( dHVG ) and in the Tahoe-x1 embedding space ( dTahoe ), and with the alignment ∣cos∣ of the observed response with the top singular direction of the held-out responses. Spearman ρ for each alone; partial Pearson r given the variable after the bar.
Figure 6: Tahoe + ST on HepG2, 945 held-out perturbations. PDC against the distance to the nearest training response in gene space and in the Tahoe-x1 embedding space (top), PDC against the alignment with the top singular direction (bottom left), and the alignment against the gene-space distance (bottom right).
Top direction
Random direction
Encoder
Cell line
L0
L1
L2
L3
L4
L5
L6
L7
L0
L1
L2
L3
L4
L5
L6
L7
SE+ST
HepG2
99
98
97
98
97
99
99
99
4
99
100
101
97
99
95
98
SE+ST
Jurkat
96
98
96
99
98
97
97
98
-3
95
94
96
96
94
96
97
SE+ST
RPE1
99
99
99
98
99
101
99
100
2
103
99
100
99
99
99
98
SE+ST
K562
102
99
98
100
99
99
99
99
-4
98
98
98
99
99
100
100
Tahoe+ST
HepG2
97
99
95
93
89
87
95
100
42
96
88
94
91
94
97
99
Appendix
Table 6: PDC at each layer when the attention output of that layer is reduced to one direction, as a percentage of the PDC without the intervention. Top direction : the top singular direction of the layer’s output. Random direction : a random direction outside the top 32 , with each row rescaled to its original norm. Values near 100 mean that the layer’s output can be reduced this way with little loss. The largest losses are for the random direction at L0 .
Figure 7: Direction swap in K562. Each row is the prediction for a perturbation that received the L0 direction of the perturbation named on the row, and each column is the prediction without the intervention for the perturbation named on the column. Entries are correlations of the responses. Six pairs per encoder that are both predicted well and have dissimilar observed responses, chosen so that the column perturbations differ from each other.
Encoder
Donor
HepG2
Jurkat
RPE1
K562
SE-600M
other cell lines
−0.139
−0.112
−0.008
−0.117
same cell line
−0.013
0.004
−0.005
−0.004
Tahoe-x1
other cell lines
−0.208
−0.159
−0.226
−0.164
same cell line
−0.001
−0.001
0.000
−0.005
scGPT
other cell lines
−0.057
−0.073
−0.007
−0.082
same cell line
0.020
−0.012
−0.003
0.036
Appendix
Table 7: Change in mean PDC when the top L0 direction of each perturbation is replaced by the one the same encoder gives it in a decoder trained on another cell line (mean over the three other cell lines). “Same cell line” rows use a direction from another run on the recipient’s own cell line, as a check of the replacement itself.
Cell line
PR
PRcen
rank 1
rank 4
rank 16
resampled
Tahoe + ST
HepG2
4.29
8.26
0.448
0.548
0.661
0.854
0.446
Jurkat
10.96
19.85
0.337
0.496
0.624
0.840
0.396
RPE1
2.60
5.37
0.566
0.631
0.736
0.923
0.558
K562
13.05
19.81
0.290
0.457
0.586
0.830
0.393
Appendix
Table 8: Rank of the observed responses of the held-out perturbations in each cell line ( 2,000 HVGs). PR : participation ratio; PRcen : after subtracting the mean response. Rank k : mean per-perturbation correlation between the observed responses and their top- k SVD approximation. Resampled: median correlation of each perturbation’s response with itself after resampling its cells. Last column: PDC of Tahoe + ST on all held-out perturbations, from the run of Appendix C.4 (Table 1 uses the perturbations scored by every predictor).
Encoder
HepG2
Jurkat
RPE1
K562
SE-600M
+0.144
+0.174
+0.163
+0.136
Tahoe-x1
+0.076
+0.074
+0.074
+0.064
scGPT
+0.159
+0.160
+0.152
+0.268
Appendix
Table 9: PDC of the five-neighbor retrieval oracle minus PDC of the decoder, on all held-out perturbations. Positive means retrieval is more accurate.
Pert
PDC
n test cells
Jaccard top-20
cos( δobs , δpred )
shared + sign /20
RPL6
0.980
106
0.67
0.962
16
RPL23A
0.975
67
0.67
0.952
16
RPL35A
0.973
174
0.54
0.943
14
RPL23
0.973
134
0.60
0.943
15
RPL7
0.971
110
0.60
0.933
15
RRS1
0.970
132
0.67
0.948
16
Appendix
Table 10: The ten held-out perturbations with the highest PDC for Tahoe + ST on HepG2. Cosine: between predicted and observed responses over the 2,000 HVGs, with the observed control mean subtracted from both. Jaccard top-20: overlap of the 20 genes with the largest predicted and observed changes. Shared + sign: genes in both top-20 lists with the same sign.
gene
loading
gene
loading
gene
loading
gene
loading
TUBA1B
1.06
STMN1
1.02
H2AFZ
0.90
MALAT1
−0.75
HMGB2
0.73
TUBB
0.72
HMGB1
0.70
CENPF
0.67
NEAT1
−0.67
HIST1H4C
0.63
PCLAF
0.62
MDM2
−0.62
MKI67
0.61
HMGN2
0.59
BIRC5
0.58
NASP
0.58
CALR
0.57
MCM3
0.56
PTMA
0.54
HSP90B1
0.52
SCD
0.52
ACTG1
0.51
TOP2A
0.51
TPX2
0.49
Appendix
Table 11: Genes with the largest loadings on the leading direction for Tahoe + ST on HepG2. The sign is that of the first principal axis of the v1 vectors and has no biological orientation.
Cell line
Top-FDR Hallmark category
padj
HepG2
E2F-Targets
4.78×10−8
RPE1
E2F-Targets
2.14×10−9
Jurkat
Myc-Targets-V1
1.04×10−5
K562
mTORC1-Signaling
2.67×10−4
Hypoxia (tied)
2.67×10−4
Appendix
Table 12: Hallmark category with the smallest adjusted p -value (MSigDB Hallmark 2020, Benjamini–Hochberg, background the 2,000 HVGs) among the 30 genes with the largest absolute loadings on the first right singular vector of each cell line’s matrix of observed responses.
percentile
cosine
Jaccard top-20
shared+sign / 20
p10
−0.034
0.000
0
p25
0.056
0.026
0
median
0.257
0.081
3
p75
0.648
0.290
9
p90
0.858
0.575
15
Appendix
Table 13: Agreement between predicted and observed responses over all 945 held-out perturbations (Tahoe + ST, HepG2), by percentile.
PDC decile
PDC med
cosine p25
cosine median
Jaccard med
1
−0.045
−0.060
−0.007
0.026
2
0.056
−0.033
0.027
0.026
3
0.142
−0.008
0.058
0.026
4
0.239
0.061
0.122
0.026
5
0.372
0.132
0.186
0.053
6
0.520
0.259
0.353
0.111
Appendix
Table 14: Agreement between predicted and observed responses by PDC decile ( ≈95 perturbations each; decile 1 has the lowest PDC).
Figure 8: RPL18 (top) and GFM1 (bottom) under Tahoe + ST on HepG2. (A, D) Cosine of each held-out perturbation’s response with the leading direction against its PDC, with the perturbation marked. (B, E) The 20 genes with the largest observed change, observed (grey) and predicted; mitochondrially encoded transcripts are labelled. (C, F) Predicted against observed change for all 2,000 HVGs, each relative to its own control mean, with their cosine and Pearson R (the PDC).
Figure 9: Per-perturbation PDC by gene family for the three encoders (columns) in the four cell lines (rows). Violins show the distribution and ticks the median; n is the number of perturbations in each family.
Encoder
Cell line
(a)
(b)
(c)
(d)
(e)
SE
HepG2
0.844
0.769
0.827
0.322
0.744
SE
Jurkat
0.832
0.692
0.819
0.498
0.652
SE
RPE1
0.817
0.822
0.797
0.707
0.712
SE
K562
0.878
0.751
0.870
0.511
0.719
Tahoe
HepG2
0.840
0.823
0.808
0.585
0.702
Tahoe
Jurkat
0.850
0.828
0.822
0.688
0.662
Appendix
Table 15: Spearman ∣ρ∣ between each score and PDC on the held-out perturbations. (a) predicted-NN-distance (cosine, nearest training response); (b) norm of the predicted response; (c) mean cosine distance to the ten nearest training responses; (d) Euclidean distance to the nearest training response; (e) alignment of the predicted response with the leading axis of the training responses.
Encoder
Cell line
ntest
unbin. ρ
Q1
Q2
Q3
Q4
Q5
MOD
STR
SE
HepG2
945
−0.844
−0.412
−0.555
−0.549
−0.669
−0.830
✓
✓
SE
Jurkat
1084
−0.832
−0.484
−0.473
−0.657
−0.835
−0.871
✓
✓
SE
RPE1
1061
−0.817
−0.395
−0.375
−0.371
−0.410
−0.647
✓
—
SE
K562
968
−0.878
−0.436
−0.593
−0.767
−0.790
−0.813
✓
✓
Tahoe-x1
HepG2
945
−0.840
−0.367
−0.536
−0.368
−0.589
−0.739
✓
—
Tahoe-x1
Jurkat
1084
−0.850
−0.483
−0.512
−0.458
−0.517
−0.741
✓
✓
Appendix
Table 16: Spearman ρ between predicted-NN-distance and PDC on all held-out perturbations (unbinned) and within five equal-count bins of the norm of the predicted response (Q1 smallest). MOD: at least 3 of 5 bins with ∣ρ∣≥0.2 ; STR: at least 4 of 5 bins with ∣ρ∣≥0.4 .
Bin
Cross-config median ∣ρ∣
IQR ( 25%/75% )
#∣ρ∣≥0.2
#∣ρ∣≥0.4
Q1
0.404
0.377/0.448
11/12
6/12
Q2
0.521
0.443/0.565
10/12
9/12
Q3
0.529
0.370/0.614
10/12
8/12
Q4
0.608
0.494/0.726
11/12
11/12
Q5
0.734
0.641/0.770
11/12
11/12
Appendix
Table 17: Within-bin ∣ρ∣ over the 12 configurations: median, interquartile range, and the number of configurations above 0.2 and 0.4 .
Figure 10: (A) Within-bin ∣ρ∣ between predicted-NN-distance and PDC in five bins of predicted norm, one marker per configuration; boxes summarize the 12 configurations; dotted and dashed lines at 0.2 and 0.4 . (B) For each configuration and bin, ∣ρ∣ of predicted-NN-distance against ∣ρ∣ of the remaining variation in predicted norm.
Cell line
target PRctrl-rel
mean stratified ∣ρ∣
mean unbinned ∣ρ∣
attenuation ratio
HepG2
4.29
0.558
0.838
0.665
Jurkat
10.96
0.608
0.845
0.719
RPE1
2.60
0.271
0.700
0.387
K562
13.05
0.644
0.842
0.765
Appendix
Table 18: Mean within-bin ∣ρ∣ (over the three encoders and five bins) and mean unbinned ∣ρ∣ in each cell line, their ratio, and the participation ratio of the observed responses (Table 8 ).
Figure 11: Mean within-bin ∣ρ∣ against the participation ratio of the observed responses in each cell line, one marker per encoder; bars give the mean over encoders.
Figure 12: Transfer of the per-configuration threshold between all 12×12 source–target pairs. Entry: mean PDC above the rank- 1 SVD baseline among predictions accepted on the target (columns) using the threshold fitted on the source (rows). Hatched: fewer than five accepted predictions. White rules separate the encoder blocks.
Figure 13: Mean PDC of the accepted predictions against the share accepted, as the threshold on predicted-NN-distance is raised (rows: encoders; columns: cell lines). Shaded: bootstrap 95% interval ( 5,000 resamples); dashed: rank-1 SVD baseline; star: the pooled threshold, labeled with its share accepted. Lower right: median and interquartile range over the 12 configurations.
Learn-then-Test
Decile rule
Encoder
Cell line
Accepted
PDC
Accepted
PDC
SE
HepG2
0.696
0.520
0.465
0.654
SE
Jurkat
0.380
0.574
0.342
0.600
SE
K562
0.506
0.553
0.419
0.603
SE
RPE1
0.987
0.475
0.612
0.651
Tahoe
HepG2
0.897
0.494
0.584
0.646
Appendix
Table 19: Learn-then-Test and the decile rule over 200 random calibration/deployment splits: share of predictions accepted and mean PDC of accepted predictions.
Exphormer
GAT
Mean PDC
0.600
0.595
Oracle distance vs PDC, ∣ρ∣
0.821
0.802
Predicted vs observed distance, ρ
0.607
0.443
Predicted-NN-distance vs PDC, ∣ρ∣
0.697
0.481
Top- 1 singular direction
0.366
0.344
Random direction, rank 1
0.031
−0.176
Appendix
Table 20: TxPert: distance score and rank- k projection of the perturbation latent (mean PDC). Exphormer checkpoints: the distance rows (top block) use K562_unseen_pert_exphormer_mg (STRING and GO graphs); the projection rows (bottom block) and the participation ratio in the text use K562_unseen_pert_exphormer (STRING graph).
Cell line
ρ
95% interval
PR observed
PR predicted
C32
−0.849
[−0.871,−0.821]
3.48
2.39
HOP62
−0.788
[−0.819,−0.749]
5.55
4.44
HepG2-C3A
−0.683
[−0.723,−0.635]
5.60
4.12
Hs 766T
−0.726
[−0.767,−0.680]
7.79
5.92
PANC-1
−0.710
[−0.750,−0.667]
7.03
5.37
Appendix
Table 21: Tahoe-100M chemical perturbations: Spearman ρ between predicted-NN-distance and per-condition PDC with drug-clustered 95% intervals, and participation ratio (PR) of observed and predicted responses.
Context size S
Encoder
Cell line
Pseudobulk
Single cell
16
32
64
128
256
SE
HepG2
0.844
0.744
0.852
0.841
0.850
0.837
0.841
SE
Jurkat
0.832
0.647
0.862
0.852
0.849
0.855
0.858
SE
K562
0.878
0.753
0.878
0.884
0.879
0.890
0.881
SE
RPE1
0.817
0.765
0.811
0.823
0.817
0.822
0.813
Tahoe
HepG2
0.840
0.712
0.870
0.855
0.841
0.833
0.839
Appendix
Table 22: Spearman ∣ρ∣ between predicted-NN-distance and PDC: pseudobulk and single-cell scoring, and pseudobulk scoring at context size S . The S=64 column is an independent rerun of the pseudobulk column; the two differ by at most 0.020 .
SE + ST
Tahoe + ST
scGPT + ST
Cell line ( n )
ρ
PDC
ρ
PDC
ρ
PDC
HepG2 ( 290 )
−0.697
0.251
−0.486
0.316
−0.588
0.316
Jurkat ( 280 )
−0.363
0.279
−0.168
0.281
−0.505
0.170
K562 ( 313 )
−0.474
0.114
−0.478
0.176
−0.464
0.133
RPE1 ( 328 )
+0.366
0.471
+0.314
0.463
+0.497
0.466
Appendix
Table 23: Strict family-held-out retraining: Spearman ρ between predicted-NN-distance and PDC on pooled held-out families ( n perturbations), and mean PDC.
Single-cell RNA foundation models (scFMs) are increasingly used for in-silico biological discovery, yet predictive performance alone does not reveal whether their decisions rely on biologically meaningful signals or model-specific shortcuts. We systematically investigate post-hoc attribution of downstream predictions across two tasks: cell-type annotation using pretrained Cell2Sentence and scGPT models, and perturbation prediction using scGPT+STATE in a controlled setting. For cell annotation, we use attribution to assess whether predictions rely on established marker genes and how their relevance varies with model architecture, expression rank, and the surrounding cellular context. For perturbation prediction, we compare attribution patterns with known regulatory ground-truth in a synthetic setting to assess what they reveal about the model's learned computations. Across these tasks, attribution reveals biological feature use, contextual interactions, contributions from different input pathways, learning progression, associations with prediction quality, and potential pitfalls that are not fully captured by standard performance metrics. Experiments with complete and limited regulatory knowledge further demonstrate that conclusions drawn from attribution depend strongly on the choice of biological reference, baseline, and null model. Together, these findings highlight the value of attribution as a flexible post-hoc diagnostic for understanding model behavior, identifying potential issues, and informing the development of scFMs.
BIFOLD - Berlin Institute for the Foundations of Learning and Data · Technical University of Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany +1
We present a systematic evaluation framework - thirty-seven analyses, 153 statistical tests, four cell types, two perturbation modalities - for assessing mechanistic interpretability in single-cell foundation models. Applying this framework to scGPT and Geneformer, we find that attention patterns encode structured biological information with layer-specific organisation - protein-protein interactions in early layers, transcriptional regulation in late layers - but this structure provides no incremental value for perturbation prediction: trivial gene-level baselines outperform both attention and correlation edges (AUROC 0.81-0.88 versus 0.70), pairwise edge scores add zero predictive contribution, and causal ablation of regulatory heads produces no degradation. These findings generalise from K562 to RPE1 cells; the attention-correlation relationship is context-dependent, but gene-level dominance is universal. Cell-State Stratified Interpretability (CSSI) addresses an attention-specific scaling failure, improving GRN recovery up to 1.85x. The framework establishes reusable quality-control standards for the field.
Ihor Kendiukhov
Department of Computer Science, University of Tübingen, Tübingen, Germany
Identifying the drug and dose that produced a cellular response requires preserving treatment effects within each cellular background. A joint-embedding predictor can match its targets while ignoring the intervention if its encoder preserves differences across backgrounds but erases differences between treated and control populations within each one. We call this failure interventional contrast collapse. PerturbJEPA encodes the difference in mean gene expression between treated and control populations as an effect vector. It aligns the displacement between their latent centroids with this vector and regularizes the vectors to preserve variation across population pairs. We show that the measured displacements cannot all coincide within a batch when effect-vector variation exceeds the root-mean-square alignment error. Trained with separate drug and dose ranking objectives, the model predicts candidate responses from a common control population and matches them to the observed response without cell pairing or expression reconstruction. We evaluate 3,592 Tahoe-100M queries on held-out combinations of known interventions and cellular backgrounds, each with over 700 drug-dose candidates in a fixed, background-specific catalog. PerturbJEPA achieves 91.46% macro joint drug-dose Recall@1, exceeding STATE by 3.40 percentage points (95% drug-cluster bootstrap interval of 1.72 to 5.07 points). Controlled diagnostics show that treatment-control differences can shrink even when representations remain globally dispersed. With the drug ranking objective fixed, adding effect-vector regularization to alignment raises joint Recall@1 from 18.57% to 87.47%.
Jiangshuan Pang, Youzhe He, Yiting Deng +5
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Key Laboratory of Brain Cell Mapping of Zhejiang Province, BGI Research, Hangzhou 310030, China. · Center for Excellence in Brain Science and Intelligence Technology (Institute of Neuroscience), Chinese Academy of Sciences