Model explanations are increasingly used as scientific evidence, yet it remains unclear what they license scientists to conclude beyond the data on which they were constructed. When a fine-grained explanation fails to replicate, the failure may be attributed to noisy measurement of its target, raising a fundamental question: does reliable measurement imply a reproducible explanation? We test this directly. We formalize prospective explanation transportability: the extent to which an explanation learned from past data is recovered in a genuinely future context. This property is distinct from feature importance and measurement reliability. Using multi-year Genomes-to-Fields maize trials, where each new year is a real future context rather than a random resample, we evaluate 16 agronomically interpretable environmental concepts across five forward splits and one prospectively locked test year. Coarse environment-level explanations transport reliably (median cosine 0.83); their locked 2024 value, 0.811, fell within a prespecified 80% bootstrap percentile interval. For grain yield, genotype-conditioned explanations have a historical median cosine of 0.30 and range from 0.006 to 0.822 across years. Reliability does not order these outcomes: the year with the highest split-half reliability had the lowest explanation transport. Within-context resampling showed that GxE explanations were reproducible within a fixed context (0.77-0.95) while exhibiting lower prospective alignment across the 2020-2023 transitions (0.01-0.36). A second phenotype transports strongly at both resolutions, and non-temporal validation inflates apparent interaction-level transport by +0.22. These results show that target reliability and within-context explanation reproducibility do not by themselves establish when a model explanation can support a reproducible scientific claim.
Figures & tables
Quantity
Historical
2024
Main-effect cosine
0.831 median
0.811
G × E cosine
0.295 median
0.271
Main–G × E gap
0.536
0.540
Main forecast
–
0.831
Main 80% bootstrap percentile interval
–
[0.755,0.942]
Main forecast outcome
–
inside interval
Table 1: Primary prospective evidence for grain yield. Historical values summarize the five strictly forward evaluations (2019–2023). The 2024 main-effect bootstrap percentile interval was locked before the successful scientific evaluation. No numerical G × E forecast was specified.
Stress test
Coarse/Main
Fine/G × E
Difference
Yield, XGBoost/SHAP
0.831
0.295
0.536
Yield, Ridge
0.588
0.227
0.361
Grain Moisture
0.907
0.859
0.048
Random-Env validation
0.793
0.512
–
Strict-Forward validation
0.831
0.295
–
Validation optimism gap
−0.039
+0.217
–
Table 2: Secondary tests of alternative explanations. All analyses in this table are secondary/post-hoc and leave the locked 2024 confirmatory analysis unchanged. Values are prospective cosine similarities unless otherwise noted.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Target year
Main (mean ± SD)
G × E (mean ± SD)
Main > G × E
2019
0.777±0.016
0.870±0.006
0/5
2020
0.826±0.011
0.029±0.026
5/5
2021
0.741±0.009
0.390±0.022
5/5
2022
0.920±0.003
0.308±0.019
5/5
2023
0.964±0.000
0.212±0.022
5/5
2024
0.809±0.008
0.265±0.008
5/5
Appendix
Table 3: Post-hoc SHAP-weight seed sensitivity. Mean ± standard deviation of weighted cosine across five XGBoost seeds, with the signed regression coefficients held fixed. The last column counts seeds in which Main > G × E.
Metric
Main
G × E
Unweighted cosine
0.789
0.145
Spearman correlation (absolute importance)
0.388
−0.697
Top-5 Jaccard (absolute importance)
0.429
0.000
SHAP-weighted sign agreement
0.831
0.549
Appendix
Table 4: Post-hoc alternative metrics in 2024. Full-data estimates; main versus genotype-conditioned response.
Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.
Nick Oh, Helen Jin
socius labs, London, United Kingdom · Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania, United States.
When a language model explains an answer it has already given, does it reuse the computation that produced the answer or reconstruct a story from the answer alone? Attribution, transportability and recoverability are each compatible with causal use without establishing it. We propose an evidence standard: pair each positive statistic with a variable specific null that removes the tested variable's identity while matching relevant nuisance dimensions as far as possible, and audit unmatched dimensions. We apply this standard to a known cause. A cue naming a wrong option raises the rate of choosing that option by 64 to 68 percentage points across three models. Explanations mention the cue in 1.8 percent of items or fewer in three of four models tested. Three estimator classes yield favorable statistics, but none establishes causal sensitivity to the cue contrast under its own control in the three-model analysis. In the strongest case, a recovered cue direction reaches R2 of 0.95 and exceeds a geometry matched random direction in all three seeds, while a direction fitted by the same pipeline with cue labels scrambled reproduces 61 to 76 percent of its effect at comparable realized edit magnitude. A fourth model passes one interchange endpoint, but unequal edit magnitudes and a contrast that changes both cue identity and cue-answer agreement limit its interpretation. These experiments leave causal access unresolved. They establish an evidentiary requirement: favorable mechanistic statistics must survive controls for variable identity and nuisance structure. Reusable controls separate generic from identity specific transport effects, fit null directions with scrambled labels, and audit realized intervention magnitudes.
Arshia Eftekhari zadeh
School of Electrical and Computer Engineering University of Tehran, Tehran, Iran
The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.