Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Angel Wang; Dominique Perrault-Joncas; Alvaro Maggiar; Dean Foster; Carson Eisenach

cs.LGOct 2, 2026

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Authors: Angel Wang, Dominique Perrault-Joncas, Alvaro Maggiar, Dean Foster, Carson Eisenach

Organizations: Amazon

Abstract

Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.

Figures & tables

Component	Calibration slope (95% CI)	$R^{2}$
Order quantity	$[0.70,\,0.71]$	$0.24$
Inbound	$[0.67,\,0.67]$	$0.47$
Inventory	$[0.90,\,0.91]$	$0.81$
Sales drain	$[0.99,\,1.00]$	$0.92$

Table 1: Component-level trajectory calibration (Study 1): realized values regressed on simulated values at the product–week level, with slope

1

indicating perfect calibration. Order and inbound exhibit larger calibration gaps; inventory and sales drain are better calibrated.

Study	Training source	1 week	2 weeks	3 weeks	4–10 weeks
Study 1	Simulator rollouts	$13.3\pm 2.7$	$11.7\pm 2.8$	$13.2\pm 3.1$	$14.8\pm 2.8$
Study 1	Historical (off-policy)	$15.0\pm 2.9$	$14.8\pm 3.1$	$15.2\pm 3.1$	$16.0\pm 3.1$
Study 2	Simulator rollouts	$12.7\pm 3.6$	$15.3\pm 7.4$	$9.4\pm 3.8$	$11.5\pm 3.4$
Study 2	Historical (off-policy)	$25.6\pm 5.2$	$27.8\pm 6.3$	$28.1\pm 6.4$	$25.2\pm 5.7$

Table 2: Zero-shot transfer: MAPE (mean

\pm

95% CI) on realized deployment outcomes for the same forecaster architecture trained on simulator rollouts under the deployed policy versus historical real data from prior policies (off-policy), by forecast horizon. CIs are 95% normal-approximation intervals across weekly forecast origins, with one error observation per origin and horizon.

Study	Variant	1 week	2 weeks	3 weeks	4–10 weeks
Study 1	Zero-shot	$13.3\pm 2.7$	$11.7\pm 2.8$	$13.2\pm 3.1$	$14.8\pm 2.8$
Study 1	+Calibration	$11.8\pm 2.6$	$11.2\pm 3.1$	$13.1\pm 3.6$	$14.8\pm 3.2$
Study 2	Zero-shot	$12.7\pm 3.6$	$15.3\pm 7.4$	$9.4\pm 3.8$	$11.5\pm 3.4$
Study 2	+Calibration	$12.6\pm 3.6$	$13.0\pm 7.1$	$6.9\pm 3.1$	$9.7\pm 2.9$

Table 3: Calibration with early real observations: MAPE (mean

\pm

95% CI) for the simulator-trained forecaster, zero-shot versus after regression calibration on early deployment observations, by forecast horizon. Zero-shot and calibrated variants are evaluated on the same forecast origins as Table 2 , whose interval definition applies. Calibrated historical-data baselines are reported in Appendix D .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Example capacity target trajectories generated by the wavelet sampler using a truncated Haar wavelet basis.

Hyperparameter	Value	Notes
Optimization
Optimizer	Adam
Learning rate	$10^{-3}$	Fixed throughout
Batch size	3,000	Products sampled per rollout; 5 cohorts per epoch
Cohort resampling	every 10 epochs	Uniform over the study population
Training epochs	2,000

Appendix

Table 4: Training hyperparameters for the response forecaster

\psi_{\theta}

Exogenous variable	Coef. (95% CI)	$R^{2}$
Demand	$[1.37,\,2.44]$	$0.02$
Price	$[-1.68,\,0.18]$	$0.08$
Cost	$[-0.26,\,0.19]$	$0.05$
Vendor lead time	$[0.35,\,0.37]$	$0.01$

Appendix

Table 5: Exogeneity check (Study 1): regression of representative exogenous variables on the treatment indicator. Under exogeneity, coefficients should be indistinguishable from zero.

	One-step		Continuous
Component	Coef. (95% CI)	$R^{2}$	Coef. (95% CI)	$R^{2}$
Order quantity	$[0.70,\,0.71]$	$0.24$	$[0.65,\,0.66]$	$0.25$
Inbound	$[0.67,\,0.67]$	$0.47$	$[0.65,\,0.66]$	$0.12$
Inventory	$[0.90,\,0.91]$	$0.81$	$[1.04,\,1.04]$	$0.87$
Sales drain	$[0.99,\,1.00]$	$0.92$	$[0.94,\,0.94]$	$0.87$

Appendix

Table 6: Rollout-protocol sensitivity (Study 1): product–week calibration under one-step versus continuous rollouts. The one-step column corresponds to the diagnostics of Table 1 .

Figure 2: Trajectory fidelity under the deployed policy: each point represents a product–week, with the $x$ -axis showing simulated values and the $y$ -axis showing corresponding realized observations; both axes of each panel are scaled to $[0,1]$ by a common constant, which leaves the regression slope and $R^{2}$ unchanged. A linear regression fit reports slope and $R^{2}$ , with the dotted line indicating perfect calibration.

Figure 3: Policy sensitivity: the $x$ -axis shows simulated values under the deployed policy, the $y$ -axis realized observations; both axes of each panel are scaled to $[0,1]$ by a common constant. Separate regressions for products operated under the deployed policy (on-policy) and under the baseline (off-policy); the dotted line indicates perfect calibration.

Training Source	Eval Source	Model	1 week	2 weeks	3 weeks	4–10 weeks
Simulator (Study 1)	Simulator (Study 1)	Global Aggregate	$9.2\pm 1.8$	$9.2\pm 1.6$	$10.8\pm 2.0$	$11.1\pm 2.0$
Simulator (Study 1)	Simulator (Study 1)	Bottom-Up	$12.2\pm 3.1$	$11.5\pm 2.7$	$10.7\pm 2.5$	$9.6\pm 0.9$
Simulator (Study 2)	Simulator (Study 2)	Global Aggregate	$10.0\pm 1.4$	$10.5\pm 1.5$	$11.9\pm 1.6$	$9.1\pm 1.3$
Simulator (Study 2)	Simulator (Study 2)	Bottom-Up	$10.0\pm 2.6$	$12.9\pm 2.8$	$14.0\pm 3.0$	$12.2\pm 1.7$
Historical Real Data	Held-out Real Data	Global Aggregate	$11.7\pm 1.5$	$11.9\pm 1.5$	$11.8\pm 1.4$	$11.5\pm 1.3$
Historical Real Data	Held-out Real Data	Bottom-Up	$8.9\pm 2.2$	$9.9\pm 2.3$	$10.5\pm 2.4$	$9.7\pm 1.2$

Appendix

Table 7: Learnability of simulator-generated dynamics. Models are trained on simulator rollouts and evaluated on held-out simulator data from the same study. Historical real-data baselines are trained on pre-study historical observations and evaluated on held-out real data from the Study 2 evaluation period, using the same model classes and training procedures. Reported values are MAPE (mean

\pm

95% CI).

Training Source	Eval Source	Model	1 week	2 weeks	3 weeks	4–10 weeks
Simulator (Study 1)	Real (Study 1)	Global Aggregate	$13.3\pm 2.7$	$11.7\pm 2.8$	$13.2\pm 3.1$	$14.8\pm 2.8$
Simulator (Study 1)	Real (Study 1)	Bottom-Up	$13.4\pm 3.0$	$13.3\pm 3.0$	$13.4\pm 2.8$	$13.4\pm 1.1$
Simulator (Study 2)	Real (Study 2)	Global Aggregate	$12.7\pm 3.6$	$15.3\pm 7.4$	$9.4\pm 3.8$	$11.5\pm 3.4$
Simulator (Study 2)	Real (Study 2)	Bottom-Up	$16.4\pm 6.2$	$17.7\pm 7.4$	$20.1\pm 8.0$	$19.4\pm 3.9$
Off-policy Real Data	Real (Study 1)	Global Aggregate	$15.0\pm 2.9$	$14.8\pm 3.1$	$15.2\pm 3.1$	$16.0\pm 3.1$
Off-policy Real Data	Real (Study 1)	Bottom-Up	$16.7\pm 2.9$	$16.8\pm 3.0$	$17.7\pm 3.0$	$19.5\pm 1.2$

Appendix

Table 8: Zero-shot transfer, full results. Models are trained on pre-evaluation simulator rollouts under the deployed policy or on historical real data collected under prior operating policies (off-policy), and evaluated on realized aggregate utilization during the matched deployment windows. Reported values are MAPE (mean

\pm

95% CI) by forecast horizon.

Training Source	Eval Source	Model	1 week	2 weeks	3 weeks	4–10 weeks
Simulator (Study 1)	Real (Study 1)	Global Aggregate + Calib.	$11.8\pm 2.6$	$11.2\pm 3.1$	$13.1\pm 3.6$	$14.8\pm 3.2$
Simulator (Study 1)	Real (Study 1)	Bottom-Up + Calib.	$13.5\pm 6.0$	$12.4\pm 5.7$	$12.3\pm 5.4$	$11.2\pm 1.8$
Simulator (Study 2)	Real (Study 2)	Global Aggregate + Calib.	$12.6\pm 3.6$	$13.0\pm 7.1$	$6.9\pm 3.1$	$9.7\pm 2.9$
Simulator (Study 2)	Real (Study 2)	Bottom-Up + Calib.	$19.6\pm 6.8$	$21.3\pm 6.3$	$18.2\pm 9.1$	$14.4\pm 4.7$
Off-policy Real Data	Real (Study 1)	Global Aggregate + Calib.	$15.5\pm 3.0$	$15.9\pm 3.3$	$15.8\pm 3.6$	$18.0\pm 3.4$
Off-policy Real Data	Real (Study 1)	Bottom-Up + Calib.	$18.4\pm 3.0$	$15.4\pm 2.6$	$15.2\pm 2.3$	$14.1\pm 1.8$

Appendix

Table 9: Available forecasting results after regression-based calibration using early real observations. Simulator-trained models are corrected using data from the beginning of the evaluation period. Off-policy historical baselines use the same calibration procedure. The retained source results do not contain every model–study combination. Values are MAPE (mean

\pm

95% CI).

Explore similar work

Mind the Sim-to-Real Gap & Think Like a Scientist
May 20, 2026Harsh Parikh, Gabriel Levin-Konigsberg, Dominique Perrault-Joncas +1Sim-to-Real Transfer RL Exploration
Amazon SCOT, Seattle, USA · Yale University, New Haven, USA · Duke University, Durham, USA
ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jun 17, 2026Jaeho Lee, Nick Merrill, Ezra KargerForecasting Benchmarks Probabilistic Forecasting
1Forecasting Research Institute · 2UC Berkeley
Sim+Real: Joint Simulation - Experiment Training Improves Balanced Prediction in Physical Systems
Oct 1, 2026Mahindra Rautela, Alexander Scheinker, Ayan Biswas +3Sim-to-Real Transfer PDE Surrogate Modeling
Los Alamos National Laboratory, New Mexico, USA

May 20, 2026, cs.AIJ/K move · Enter open · S save

Mind the Sim-to-Real Gap & Think Like a Scientist

Harsh Parikh, Gabriel Levin-Konigsberg, Dominique Perrault-Joncas, Alexander Volfovsky

Amazon SCOT, Seattle, USA · Yale University, New Haven, USA · Duke University, Durham, USA

Sim-to-Real Transfer RL Exploration Bayesian Experimental Design

Open paper AskPDF arXiv

Suppose a planner has a pre-trained simulator of a sequential decision problem and the option to run real experiments in the field. The simulator is cheap to query but inherits confounding and drift from its calibration data. Experimentation is unbiased but consumes one real unit per trial. We study when, and how, the planner should supplement the simulator with experiments. We give three results. First, an extended simulation lemma decomposes the simulator's value error into a calibration--deployment shift that randomization can identify and a parametric residual that no further interaction can reduce. Second, the value gap between the simulator-optimal policy and the optimum splits into a local component, on states the deployed policy already visits, and a reachability component, on states it does not. The reachability component stays bounded away from zero at any horizon under purely passive learning. Third, we propose Fisher-SEP, a simulation-aided experimental policy (SEP) that minimizes the posterior predictive variance of a target policy's value, with reward-only and transition-only specializations. Two case studies illustrate the regimes. In a vending-machine supply chain, front-loaded experimentation overtakes posterior updating once the horizon is long enough to amortize the pilot. In an HIV mobile-testing example with a corridor that separates a well-surveilled region from a poorly-surveilled one, only designed exploration reaches the poorly-surveilled region.

Abstract

Explore similar work

Mind the Sim-to-Real Gap & Think Like a Scientist

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

Sim+Real: Joint Simulation - Experiment Training Improves Balanced Prediction in Physical Systems