Counterfactual Predictions in Scientific Emulators Without Controlled Experiments

Dingling Yao; Kahaan Gandhi; Valentin Duruisseaux; Boris Bonev; Francesco Locatello; Anima Anandkumar

cs.LGSep 30, 2026

Counterfactual Predictions in Scientific Emulators Without Controlled Experiments

Authors: Dingling Yao, Kahaan Gandhi, Valentin Duruisseaux, Boris Bonev, Francesco Locatello, Anima Anandkumar

Organizations: Institute of Science and Technology Austria · California Institute of Technology · NVIDIA

Abstract

Many scientific questions require reasoning about what was never observed: What if the conditions, interventions, or history had been different? Models can predict accurately on observed data yet fail on such what-if queries when correlated inputs are varied independently. A common remedy is to add controlled simulation data in which these factors are explicitly disentangled, but this requires access to a simulator, can be computationally expensive, and inherits the simulator's modeling assumptions. We introduce ReRoute, a framework for targeted scientific what-if prediction that combines factual data with partial mechanistic knowledge, without requiring controlled intervention data for adaptation. ReRoute fixes the queried input of a pretrained backbone to a reference value, reintroduces its variation through a known mechanistic pathway, and fine-tunes on the original factual data, while leaving downstream effects to the learned dynamics. We provide a causal identification result for this construction under explicit structural assumptions, with the core argument machine-checked in Lean. After showing that ReRoute achieves highly accurate counterfactual predictions in a controlled advection-diffusion system where exact responses are available, we turn to state-of-the-art climate emulation. On held-out coupled-climate interventions, ReRoute reduces aggregate climate error by 18.2-31.8% under severe CO $_2$ distribution shifts while preserving skill under standard conditions, at a small fraction of the cost of retraining on additional controlled simulations, without even accounting for the substantial expense of generating such data. Finally, on an emulator trained from historical ERA5 reanalysis, where no counterfactual reference exists, ReRoute preserves substantially more of the surface warming implied by the observed boundary conditions under a fixed-CO $_2$ counterfactual.

Figures & tables

Figure 1: Overview of the ReRoute framework. (a) Standard training: the emulator

G_{\theta}

is trained on factual samples

(\mathbf{x},\mathbf{s},\mathbf{z},\mathbf{y})

. (b) ReRoute initializes from the pretrained operator

G_{\theta}

, replaces the targeted input

\mathbf{s}

seen by the backbone with the fixed reference

\mathbf{s}_{0}

, and reintroduces the factual value

\mathbf{s}

only through the fixed known pathway

\Gamma(\mathbf{s}_{0},\mathbf{s})

. The model is fine-tuned using the same factual data

(\mathbf{x},\mathbf{s},\mathbf{z},\mathbf{y})

. (c) For a what-if query

\mathbf{s}^{\prime}

, the backbone remains conditioned on

\mathbf{s}_{0}

, while

\mathbf{s}^{\prime}

enters through the pathway

\Gamma(\mathbf{s}_{0},\mathbf{s}^{\prime})

, producing the predicted response under the requested value.

Figure 2: ReRoute remains accurate over long autoregressive rollouts. Counterfactual-response RMSE over 380 held-out parameter pairs far from the training diagonal, averaged across held-out initial conditions. Fine-tuning uses 10-step rollouts; evaluation extends to 100 steps.

Figure 3: ReRoute becomes increasingly beneficial under severe SST–CO 2 distribution shift. We compare ACE2S-SHiELD+ no-RC (Baseline) with the same pretrained checkpoint augmented with ReRoute . Bars show the mean per-realization aggregate climate error

\alpha

, with error bars denoting one standard deviation across independent ensembles. Performance remains comparable under standard AMIP and the milder fixed-CO 2 intervention, while ReRoute reduces error by

18.2\%

under abrupt-

4\times

CO 2 and by

31.8\%

under the held-out random-CO 2 forcing. The abrupt-

4\times

CO 2 experiment uses 36 initial-condition members; the remaining experiments use five stochastic realizations.

Figure 4: ReRoute preserves surface warming under fixed CO 2 in ACE2-ERA5, i.e. near-zero values in the heatmap. Shown is the change in time-mean near-surface temperature when CO 2 is held fixed at its 1940 value instead of following the historical trajectory, while retaining the other historical boundary conditions such as SST and sea-ice. The ACE2-ERA5 baseline exhibits widespread cooling, indicating that much of its historical warming is spuriously tied to the CO 2 input. With ReRoute , the spurious cooling is substantially corrected over most of the globe, showing that the model retains the surface warming driven by the prescribed boundary conditions.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Roadmap of the identification argument.

Figure 6: ReRoute separates strongly correlated advection and diffusion controls across the full parameter grid. The left panel shows the 512 parameter pairs used for training; dashed and dotted lines denote their center and range. The remaining panels show mean counterfactual-response RMSE after 100 rollout steps for the pretrained baseline, ReRoute applied only to diffusivity $D$ , and ReRoute applied jointly to $(c,D)$ . Errors are averaged across eight held-out initial conditions. ReRoute reduces error away from the training support, with the joint pathway giving the lowest errors.

Figure 7: ReRoute removes most of the structured counterfactual-response error for a representative held-out field. Shown is the signed response residual ( in linear and log scales ) after 100 rollout steps for one held-out initial condition at $c=0.5$ and $D=0.018$ , a parameter combination far from the correlated training support. The baseline has RMSE $0.1182$ . Rerouting $D$ alone reduces this to $0.0108$ , while jointly rerouting $(c,D)$ reduces it to $0.0079$ .

Split	Simulation samples	Years
Training
AMIP	IC1: 1979–2008; IC2: 1981-07–1986-06, 1991-07–1996-06, 2001-07–2006-06	45
Equilibrium	1 $\times$ CO 2 : IC1: 2031–2040, IC2: 2032–2036 2 $\times$ CO 2 : IC1: 2031–2040, IC2: 2032–2036 4 $\times$ CO 2 : IC1: 2031–2040, IC2: 2032–2036	45
Validation
AMIP	IC1: 2009–2011	3
Equilibrium	IC2: 2038 at each of 1 $\times$ , 2 $\times$ , and 4 $\times$ CO 2	3

Appendix

Table 1: Training and validation data for ACE2S-SHiELD+

{}_{\mathrm{no\text{-}RC}}

. Time bounds are inclusive; IC denotes initial-condition ensemble member.

Prognostic variables (input and output)
Symbol	Description	Units	Time
$T_{k}$	Air temperature	K	Snapshot
$q^{T}_{k}$	Specific total water (vapor + condensates)	kg kg -1	Snapshot
$u_{k}$	Eastward wind	m s -1	Snapshot
$v_{k}$	Northward wind	m s -1	Snapshot
$T_{s}$	Skin temperature over land or sea ice	K	Snapshot

Appendix

Table 2: ACE2-ERA5 variable interface, which is also shared by ACE2S-SHiELD+ except for the differences described in § C.1 . For vertically resolved variables,

k=0,\ldots,7

indexes the eight atmospheric layers from the top of the atmosphere toward the surface. “Snapshot” denotes an instantaneous field, “Mean” a mean over the six-hour timestep, and “Invariant” a time-independent field. TOA denotes top of atmosphere.

Figure 8: ReRoute improves climatology of tropical cyclones in ACE2-ERA5. Tropical cyclone tracks detected in ERA5, the pretrained ACE2-ERA5 baseline, and ACE2-ERA5 + ReRoute over the held-out evaluation period. The same detection and tracking procedure of tropical cyclones are applied to all three datasets. ReRoute produces a spatial distribution of cyclone activity that more closely matches the ERA5 reference than the pretrained baseline.

Counterfactual Prediction

Counterfactual prediction estimates outcomes under hypothetical interventions using observed data and assumptions about how systems generate outcomes.

See all 38 papers

Aug 12, 2026

How Can Driving World Models Do Counterfactual Prediction?

Driving world models are often interpreted as counterfactual simulators for observed driving episodes: given a factual driving log, they are asked what would have happened under an alternative ego action. In this paper, we identify a fundamental mismatch between this goal and direct action-conditioned prediction. The direct prediction uses the shared history and the alternative action but not the factual continuation observed after that history. It can therefore generate a plausible future without preserving what actually happened in this episode. We formalize this gap using the causal recipe of abduction, action, and prediction and study it in a setting with a short time horizon, where the alternative ego action does not alter how surrounding agents evolve. To make the gap measurable, we construct a controlled simulation benchmark with factual outcomes and matched counterfactual outcomes. Across two representative world models, direct predictions fail to match the counterfactual ground truth, supporting our analysis. As a constructive check of this analysis, we introduce a deliberately simple, training-free pipeline that moves observed evidence into the counterfactual view and lets the frozen model complete what remains unknown. Even this simple construction raises the overall recovered fraction substantially and reduces perceptual distance to the matched counterfactual on both models. We hope this work draws attention to this gap and motivates better counterfactual prediction methods for driving world models.

Jiaru Zhang, Can Cui, Yi Xu +3

Purdue University · Bosch Center for Artificial Intelligence

Counterfactual Evaluation Causal Counterfactual Generation Counterfactual Prediction World Models for Autonomous Driving Causal Inference

Ask PDF

Aug 28, 2026

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs

Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present

\textbf{WhatIfBench}

, a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose

\textbf{PRISM}

, which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at WhatIfBench.

Yucheng Wang, Yuetian Du, Zhengyi Liu +8

Zhejiang University · Alibaba Group · City University of Hong Kong

Counterfactual Reasoning in Language Models Counterfactual Reasoning Counterfactual Evaluation Causal Reasoning in Language Models Faithfulness of Language Model Explanations

Ask PDF

Oct 2, 2026

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.

Angel Wang, Dominique Perrault-Joncas, Alvaro Maggiar +2

Amazon

Sim-to-Real Transfer Time Series Forecasting Counterfactual Prediction

Ask PDF

Abstract

Counterfactual Prediction

Explore similar work

How Can Driving World Models Do Counterfactual Prediction?

The Illusion of What If\textit{What If}What If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs