Where Should the Code Go? Locating Foundation-Model Errors in Bitemporal Inventory Simulation
Organizations: Distyl AI
Abstract
Operational reports carry two clocks: when an event happened and when its report arrived; later reports can revise earlier ones. A foundation-model component that maintains the state supported by the reports visible at a cutoff, then rolls a supplied scenario forward, can be helped by exact code before it, after it or inside the call, or asked to reason harder. We ask which remedy removes which error, for which models, on an exactly solvable lost-sales inventory environment: 36,672 independently rescored requests over twelve configurations of closed and open-weight models, with a forced-answer-call interface as a control and protocols frozen before the calls. Weak models fail in separable ways. Canonical input removes report-selection error, exact continuation removes rollout error, neither repairs the other, and which error dominates differs by model; strong models are at ceiling on raw reports. Asking for more reasoning is no substitute and can depend on the interface: under a forced answer call high effort collapses GPT-5.4 mini and DeepSeek V4 Pro, which gain or hold under automatic choice; on its provider the forced call disables reasoning for DeepSeek V4 Flash; and at high effort gpt-oss exhausts its output budget. The contribution is a controlled component diagnostic, released with every request, and evidence that for some model and provider pairs the tool-calling interface confounds reasoning evaluation.
Figures & tables
| Synthetic | Retail profiles | ||||||
|---|---|---|---|---|---|---|---|
| Model (automatic tool choice) | raw | canonical | hybrid | snapshot | raw | canonical | snapshot |
| GPT-5.4 mini | 56.8 | 86.5 | 94.8 | 94.8 | 14.6 | 82.3 | 89.6 |
| Haiku 4.5 | 52.6 | 85.9 | 92.7 | 92.7 | 6.2 | 89.6 | 92.7 |
| Haiku 4.5 + thinking | 88.5 | 96.9 | 100.0 | 99.0 | 40.6 | 91.7 | 96.9 |
| gpt-oss-20b † | 24.0 | 51.0 | 71.4 | 82.8 | 0.0 | 17.7 | 77.1 |
| gpt-oss-120b † | 49.5 | 73.4 | 76.0 | 94.8 | 5.2 | 72.9 | 86.5 |
| Forced answer call | Automatic tool choice | ||||||
|---|---|---|---|---|---|---|---|
| Model | low | high | high low | low | high | high low | Interaction |
| GPT-5.4 mini | 76.0 | 34.9 | -41.1 | 86.5 | 93.2 | +6.8 | +47.9 [+35.9, +59.9] |
| GPT-5.4 | 98.4 | 94.8 | -3.6 | 98.4 | 100.0 | +1.6 | +5.2 [+1.0, +9.4] |
| gpt-oss-20b | 52.1 | 66.1 | +14.1 | 51.0 | 59.4 | +8.3 | -5.7 [-15.6, +4.2] |
| gpt-oss-120b | 69.3 | 66.1 | -3.1 | 73.4 | 68.8 | -4.7 | -1.6 [-10.4, +6.8] |
| Qwen3.8-27B | 91.7 | 91.7 | +0.0 | 99.5 | 92.7 | -6.8 | -6.8 [-14.1, +0.5] |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Raw | Canonical | Snapshot |
|---|---|---|---|
| GPT-5.4 mini (low) | 32.8 / 37.5 | 61.5 / 76.0 | 89.6 / 89.6 |
| GPT-5.4 (low) | 83.9 / 88.0 | 94.8 / 98.4 | 99.0 / 99.0 |
| Sonnet 4.6 | 97.9 / 99.0 | 99.0 / 99.5 | 100.0 / 100.0 |
| Sonnet + thinking | 96.9 / 97.9 | 100.0 / 100.0 | 100.0 / 100.0 |
| Configuration | Raw | Canonical | Snapshot |
|---|---|---|---|
| GPT-5.4 mini (low) | 4.2 / 12.5 | 40.6 / 46.9 | 89.6 / 89.6 |
| GPT-5.4 (low) | 88.5 / 90.6 | 91.7 / 91.7 | 99.0 / 99.0 |
| Sonnet 4.6 | 91.7 / 95.8 | 97.9 / 99.0 | 100.0 / 100.0 |
| Sonnet + thinking | 96.9 / 97.9 | 100.0 / 100.0 | 100.0 / 100.0 |
| Stratum | Stock exact | Future joint | ||
|---|---|---|---|---|
| raw | canonical | raw | canonical | |
| immediate reporting | 63/96 | 86/96 | 46/96 | 71/96 |
| delayed reporting | 34/96 | 83/96 | 26/96 | 75/96 |
| no corrections | 62/96 | 91/96 | 46/96 | 76/96 |
| corrections present | 35/96 | 78/96 | 26/96 | 70/96 |
| 12 events | 50/64 | 61/64 | 43/64 | 59/64 |
| Cohort | Configuration | Arm | N | Invalid | Missing | All five | Stock | Hist. joint | Future joint |
|---|---|---|---|---|---|---|---|---|---|
| Synthetic | GPT-5.4 mini (low) | raw | 192 | 0 | 0 | 32.8 | 50.5 | 43.2 | 37.5 |
| Synthetic | GPT-5.4 mini (low) | canonical | 192 | 0 | 0 | 61.5 | 88.0 | 69.3 | 76.0 |
| Synthetic | GPT-5.4 mini (low) | snapshot | 192 | 0 | 0 | 89.6 | 100.0 | 100.0 | 89.6 |
| Synthetic | GPT-5.4 mini (low) | state-raw | 192 | 0 | 0 | 43.2 | 55.2 | 43.2 | 55.2 |
| Synthetic | GPT-5.4 mini (low) | state-canonical | 192 | 0 | 0 | 73.4 | 93.8 | 73.4 | 93.8 |
| Synthetic | GPT-5.4 mini (low) | tool-raw | 192 | 0 | 0 | 90.1 | 94.8 | 91.1 | 91.7 |
| Configuration | K-chg | K-both | D-same | D-both |
|---|---|---|---|---|
| GPT-5.4 mini (low) | 97.1 | 8.8 | 27.1 | 25.0 |
| GPT-5.4 (low) | 100.0 | 73.5 | 68.8 | 68.8 |
| Sonnet 4.6 | 100.0 | 88.2 | 89.6 | 89.6 |
| Sonnet + thinking | 100.0 | 97.1 | 93.8 | 93.8 |
| Configuration | Original repeats | Shuffled repeats | Cross-view |
|---|---|---|---|
| GPT-5.4 mini (low) | 25.0 / 20.8 | 33.3 / 33.3 | 27.1 / 25.0 |
| GPT-5.4 (low) | 70.8 / 70.8 | 70.8 / 70.8 | 68.8 / 68.8 |
| Sonnet 4.6 | 95.8 / 95.8 | 83.3 / 83.3 | 89.6 / 89.6 |
| Sonnet + thinking | 91.7 / 91.7 | 95.8 / 95.8 | 93.8 / 93.8 |
| Configuration | Input | Synthetic | Retail profiles |
|---|---|---|---|
| GPT-5.4 mini (low) | raw | 37.5 / 50.5 | 12.5 / 14.6 |
| GPT-5.4 mini (low) | canonical | 76.0 / 88.0 | 46.9 / 90.6 |
| GPT-5.4 mini (low) | snapshot | 89.6 / 100.0 | 89.6 / 100.0 |
| GPT-5.4 (low) | raw | 88.0 / 91.1 | 90.6 / 90.6 |
| GPT-5.4 (low) | canonical | 98.4 / 99.5 | 91.7 / 100.0 |
| GPT-5.4 (low) | snapshot | 99.0 / 100.0 | 99.0 / 100.0 |
| Comparison (right minus left) | Metric | Bases | Difference | 95% interval |
|---|---|---|---|---|
| GPT-5.4 mini (low)/raw canonical | future joint | 96 | +38.5 | [+28.6, +47.9] |
| GPT-5.4 mini (low)/raw canonical | stock exact | 96 | +37.5 | [+28.6, +46.4] |
| GPT-5.4 mini (low)/raw snapshot | future joint | 96 | +52.1 | [+43.8, +60.4] |
| GPT-5.4 mini (low)/raw snapshot | stock exact | 96 | +49.5 | [+41.1, +58.3] |
| GPT-5.4 (low)/raw canonical | future joint | 96 | +10.4 | [+5.7, +15.6] |
| GPT-5.4 (low)/raw canonical | stock exact | 96 | +8.3 | [+3.6, +13.5] |
| Comparison (right minus left) | Metric | Bases | Difference | 95% interval |
|---|---|---|---|---|
| GPT-5.4 mini (low)/raw canonical | future joint | 48 | +34.4 | [+24.0, +44.8] |
| GPT-5.4 mini (low)/raw canonical | stock exact | 48 | +76.0 | [+66.7, +84.4] |
| GPT-5.4 mini (low)/raw snapshot | future joint | 48 | +77.1 | [+68.8, +85.4] |
| GPT-5.4 mini (low)/raw snapshot | stock exact | 48 | +85.4 | [+78.1, +91.7] |
| GPT-5.4 (low)/raw canonical | future joint | 48 | +1.0 | [-6.2, +8.3] |
| GPT-5.4 (low)/raw canonical | stock exact | 48 | +9.4 | [+4.2, +15.6] |
| Model | Configuration | Arm | Future joint | Missing | Truncated | Mean reasoning tok. | Retried calls |
|---|---|---|---|---|---|---|---|
| GPT-5.4 mini | low, forced | raw | 37.5 | 0 | 0 | 790 | |
| low, forced | canonical | 76.0 | 0 | 0 | 515 | ||
| high, forced | raw | 25.0 | 0 | 0 | 1214 | ||
| high, forced | canonical | 34.9 | 0 | 0 | 708 | ||
| low, auto | raw | 56.8 | 1 | 0 | 975 | ||
| low, auto | canonical | 86.5 | 0 | 0 | 565 |
| Cohort | Configuration | Arm | refusal | text only | other tool | multiple calls | stream error | truncated | empty |
|---|---|---|---|---|---|---|---|---|---|
| Synthetic | DeepSeek V4 Flash (high, forced call) | canonical | 0 | 0 | 0 | 11 | 0 | 4 | 0 |
| Synthetic | DeepSeek V4 Flash (high, forced call) | raw | 0 | 0 | 0 | 8 | 0 | 2 | 0 |
| Synthetic | DeepSeek V4 Flash (low, forced call) | canonical | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| Synthetic | DeepSeek V4 Flash (low, forced call) | raw | 0 | 0 | 0 | 4 | 0 | 0 | 0 |
| Synthetic | DeepSeek V4 Pro (high, forced call) | canonical | 0 | 0 | 0 | 0 | 0 | 1 | 31 |
| Synthetic | DeepSeek V4 Pro (high, forced call) | raw | 0 | 0 | 0 | 0 | 0 | 9 | 7 |
| Cohort | Model, effort | Arm | Automatic forced (future joint) | Automatic forced (stock exact) |
|---|---|---|---|---|
| Synthetic | GPT-5.4 mini, low | raw | +19.3 [+11.5, +27.1] | +8.3 [+1.6, +15.1] |
| Synthetic | GPT-5.4 mini, high | raw | +54.2 [+44.3, +63.5] | +40.6 [+31.8, +49.5] |
| Synthetic | GPT-5.4 mini, low | canonical | +10.4 [+3.1, +17.7] | +3.6 [-1.6, +8.9] |
| Synthetic | GPT-5.4 mini, high | canonical | +58.3 [+49.5, +66.7] | +40.6 [+31.8, +49.5] |
| Synthetic | GPT-5.4, low | raw | +10.4 [+5.2, +16.1] | +7.3 [+2.6, +12.5] |
| Synthetic | GPT-5.4, high | raw | +4.7 [+1.0, +8.3] | +2.6 [+0.0, +5.2] |
| Forced answer call | Automatic tool choice | ||||||
|---|---|---|---|---|---|---|---|
| Model | low | high | high low | low | high | high low | Interaction |
| GPT-5.4 mini | 37.5 | 25.0 | -12.5 | 56.8 | 79.2 | +22.4 | +34.9 [+22.4, +46.9] |
| GPT-5.4 | 88.0 | 94.3 | +6.2 | 98.4 | 99.0 | +0.5 | -5.7 [-12.5, +1.0] |
| gpt-oss-20b | 23.4 | 42.2 | +18.8 | 24.0 | 39.1 | +15.1 | -3.6 [-12.0, +4.7] |
| gpt-oss-120b | 45.8 | 53.6 | +7.8 | 49.5 | 49.0 | -0.5 | -8.3 [-16.7, -0.5] |
| Qwen3.8-27B | 76.6 | 80.2 | +3.6 | 95.3 | 88.5 | -6.8 | -10.4 [-20.3, -0.5] |
| Raw input | Canonical input | Snapshot | |||||
|---|---|---|---|---|---|---|---|
| Cohort, configuration | model | hybrid | inside | model | hybrid | inside | model |
| Synthetic, GPT-5.4 mini (low) | 37.5 | 55.2 | 91.7 | 76.0 | 93.8 | 97.9 | 89.6 |
| Synthetic, GPT-5.4 (low) | 88.0 | 91.7 | 100.0 | 98.4 | 98.4 | 97.4 | 99.0 |
| Synthetic, Sonnet 4.6 | 99.0 | 100.0 | – | 99.5 | 100.0 | – | 100.0 |
| Synthetic, Sonnet + thinking | 97.9 | 100.0 | – | 100.0 | 100.0 | – | 100.0 |
| Retail profiles, GPT-5.4 mini (low) | 12.5 | 17.7 | 89.6 | 46.9 | 99.0 | 100.0 | 89.6 |
| Synthetic | Retail profiles | ||||||
| Model | Arm | low | medium | high | low | medium | high |
| GPT-5.4 mini, forced call | Raw | 37.5 | 26.6 | 25.0 | 12.5 | 16.7 | 20.8 |
| Canonical | 76.0 | 46.9 | 34.9 | 46.9 | 38.5 | 29.2 | |
| Snapshot | 89.6 | 97.9 | 97.4 | 89.6 | 94.8 | 93.8 | |
| GPT-5.4, forced call | Raw | 88.0 | 95.3 | 94.3 | 90.6 | 95.8 | 99.0 |
| Canonical | 98.4 | 95.8 | 94.8 | 91.7 | 94.8 | 88.5 | |