Auditing World-Model Explanations in Retail Forecasting: Faithfulness, Accuracy and Price Response
Abstract
Forecasting models that read the price and promotion plan explain each forecast as a baseline, what would sell with nothing planned, plus the lift the plan adds, and merchants order stock and plan promotions on that lift. We show that an explanation can pass the checks used to trust it and still be wrong enough to cost money. We define eight checks for any forecaster that states a plan lift, six with a pass rule fixed before any result, and run them on a world model and on LightGBM. In generated store data with known effects, the world model passes the checks of construction, stability, price direction and agreement with sales and with LightGBM, yet when merchants plan into high-demand weeks it states 1.33 to 1.40 times the true lift, and 1.23 to 1.42 times in eight variants of that world; LightGBM overstates it too (1.26, and 1.16 to 1.35). Both still pick the right weeks to promote, but stock ordered to the world model's lift is off by 0.44 to 0.48 of the true lift. The world model's stated response to a lasting regular-price change is 4.7 to 6.5 times the true -0.5, an error in kind that only a measured cause finds. Sanity checks for explanations test whether an explanation depends on the model; a decision needs to know whether it is right about the world, and these results show the first does not give the second. For the final tuned world models of our companion paper on nine data sets, baseline and lift add up under each model's own plan on every window, but a deeper price cut lowers the forecast in 29% to 83% of item-week steps on Dominick's. Across 703 configurations from the settings searches, checks passed were not related to accuracy (Spearman 0.02, 95% interval -0.02 to 0.05). A noisy benchmark of price episodes in Dominick's data is reported as a check of direction only.
Figures & tables
| Check | Passes when | Applies to |
|---|---|---|
| C1 | every violation | lift readout models |
| C2 | after the lift readout | neural models |
| C3 | mean seed | configurations with seeds |
| C4 | share | all |
| C5 | bias in and rank | all |
| C6 | non-LightGBM |
| Effect on expected sales | Size |
|---|---|
| Temporary cut, elasticity on price/regular | |
| Regular price, elasticity on regular/list | |
| Promotion | |
| Display | |
| Promotion and display together, extra | |
| Week after a planned week with nothing planned |
| Each week | Episode median | Episodes | Scored | Slope |
|---|---|---|---|---|
| 2% | 4% | 64 | 45 | |
| 2% | 5% | 55 | 38 | |
| 2% | 7% | 38 | 26 | |
| 3% | 4% | 63 | 45 | |
| 3% | 5% | 54 | 38 | |
| 3% | 7% | 37 | 26 |
| Data | Windows | C1 (audit plan) | C1 (own plan) | C2 | C3 | C4 | C5 | C6 |
|---|---|---|---|---|---|---|---|---|
| Analgesics | 6 | 0 | 100 | 83 | 83 | 0 | 83 | 0 |
| Bottled juice | 6 | 17 | 100 | 33 | 100 | 0 | 83 | 33 |
| Cheese | 6 | 0 | 100 | 100 | 100 | 0 | 83 | 17 |
| Frozen entrees | 6 | 0 | 100 | 83 | 83 | 0 | 83 | 83 |
| Soft drinks | 6 | 0 | 100 | 100 | 100 | 0 | 67 | 67 |
| Snack crackers | 6 | 0 | 100 | 83 | 100 | 0 | 67 | 33 |
| Data | Versions | Windows | C1 | C2 | C3 | C4 | C5 | C6 | All |
|---|---|---|---|---|---|---|---|---|---|
| M5 | 31 | 31 | 31/31 | 22/31 | 3/27 | 0/31 | 0/31 | 0/6 | 0/31 |
| Dominick’s | 279 | 540 | 68/540 | 423/540 | 209/224 | 16/540 | 345/540 | 5/18 | 0/540 |
| Frat | 184 | 362 | 38/362 | 325/362 | 362/362 | 326/362 | 362/362 | 6/6 | 35/362 |
| Version and window | Version | ||||||
|---|---|---|---|---|---|---|---|
| Score | M5 | Dom. | Frat | Pooled | Pooled | ||
| Checks passed | 0.205 | 0.004 | |||||
| C1 violation | 0.007 | 0.664 | |||||
| C2 agreement after randomising | 0.740 | 0.842 | |||||
| C3 seed agreement | 0.010 | 0.938 | |||||
| C4 share lower | 0.159 | 0.916 | |||||
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Zone | Store | Weeks | Price | Controls | 8 weeks | 95% interval | Full |
|---|---|---|---|---|---|---|---|---|
| Analgesics | 2 | 50 | 190–215 | 8.8 | 16 | 4.1 | 0.6 to 7.6 | 5.7 |
| Analgesics | 2 | 49 | 197–209 | 8.5 | 16 | 2.7 | 5.6 to 0.3 | 3.2 |
| Analgesics | 2 | 48 | 203–218 | 8.1 | 11 | 5.0 | 9.8 to 0.2 | 7.1 |
| Analgesics | 7 | 109 | 203–218 | 6.0 | 2 | 8.2 | 4.1 to 20.4 | 9.1 * |
| Analgesics | 1 | 32 | 204–218 | 6.5 | 7 | 13.6 | 6.7 to 20.6 | 14.9 |
| Analgesics | 2 | 88 | 204–218 | 6.7 | 10 | 3.5 | 9.1 to 2.2 | 2.0 |
| Lift against truth | |||||||||
| Model | rank | sum | price | promotion | display | C4 | C5 | Passed | RMSSE |
| Plans at random | |||||||||
| World model, seed 1 | 0.99 | 1.14 | 1.13 | 1.08 | 1.00 | 0.0% | 1.08 | 6/6 | 0.401 |
| World model, seed 2 | 0.99 | 1.10 | 1.09 | 1.06 | 1.02 | 0.0% | 1.06 | 6/6 | 0.401 |
| World model, seed 3 | 0.99 | 1.11 | 1.11 | 1.04 | 0.91 | 0.0% | 1.05 | 5/6 | 0.398 |
| World model, blind to price | 0.59 | 0.73 | 0.01 | 2.00 | 0.92 | 31.5% | 0.90 | 3/5 | 0.510 |
Explainable Time Series Forecasting
Explainable time series forecasting makes predictions over time interpretable through attributions, structured models, causal reasoning, and counterfactuals.
See all 65 papers