cs.LGNov 19, 2026

Measurable Does Not Imply Transportable: A Prospective Audit of Model Explanations in Multi-Year Field Trials

Authors: Rayhaneh Shabani Nia, Ali Karkehabadi

Organizations: University of California, Davis

Abstract

Model explanations are increasingly used as scientific evidence, yet it remains unclear what they license scientists to conclude beyond the data on which they were constructed. When a fine-grained explanation fails to replicate, the failure may be attributed to noisy measurement of its target, raising a fundamental question: does reliable measurement imply a reproducible explanation? We test this directly. We formalize prospective explanation transportability: the extent to which an explanation learned from past data is recovered in a genuinely future context. This property is distinct from feature importance and measurement reliability. Using multi-year Genomes-to-Fields maize trials, where each new year is a real future context rather than a random resample, we evaluate 16 agronomically interpretable environmental concepts across five forward splits and one prospectively locked test year. Coarse environment-level explanations transport reliably (median cosine 0.83); their locked 2024 value, 0.811, fell within a prespecified 80% bootstrap percentile interval. For grain yield, genotype-conditioned explanations have a historical median cosine of 0.30 and range from 0.006 to 0.822 across years. Reliability does not order these outcomes: the year with the highest split-half reliability had the lowest explanation transport. Within-context resampling showed that GxE explanations were reproducible within a fixed context (0.77-0.95) while exhibiting lower prospective alignment across the 2020-2023 transitions (0.01-0.36). A second phenotype transports strongly at both resolutions, and non-temporal validation inflates apparent interaction-level transport by +0.22. These results show that target reliability and within-context explanation reproducibility do not by themselves establish when a model explanation can support a reproducible scientific claim.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

    Jun 28, 2026Nick Oh, Helen JinInterpretable MLScientific ML

  2. What Would Falsify It? A Variable Specific Evidence Standard for Mechanistic Claims About Self Explanation

    Sep 26, 2026Arshia Eftekhari zadehLLM InterpretabilityFaithfulness of Language Model Explanations