Forecasting Supply Chain Disruptions with Foresight Learning

Benjamin Turtel; Paul Wilczewski; Kris Skotheim

cs.LGApr 1, 2026

Forecasting Supply Chain Disruptions with Foresight Learning

Authors: Benjamin Turtel, Paul Wilczewski, Kris Skotheim

Organizations: Lightning Rod Labs

Abstract

Anticipating supply chain disruptions before they materialize is a core challenge for firms and policymakers alike. A key difficulty is learning to reason reliably about infrequent, high-impact events from noisy and unstructured inputs - a setting where general-purpose models struggle without task-specific adaptation. We introduce an end-to-end framework that trains LLMs to produce calibrated probabilistic forecasts using realized disruption outcomes as supervision. The resulting model substantially outperforms strong baselines - including GPT-5 - on accuracy, calibration, and precision. We also show that training induces more structured and reliable probabilistic reasoning without explicit prompting. These results suggest a general pathway for training domain-specific forecasting models that produce decision-ready signals. To support transparency we open-source the evaluation dataset used in this study. Dataset: https://huggingface.co/datasets/LightningRodLabs/supply-chain-predictions

Figures & tables

Metric	Training	Test
Time span	1/2022 - 9/2025	10/2025 - 1/2026
# Countries	25	25
# Products	88	88
# Forecasting questions	4,972	452
Event rate (%)	14.9%	10.4%

Table 1: Dataset Summary Statistics

Model	Brier $\downarrow$	Brier Skill Score $\uparrow$	ECE $\downarrow$	Precision $\uparrow$
Trained model	0.0791	16.9%	0.0525	0.3478
GPT-5	0.1203	$-26.4$ %	0.1304	0.0870
GPT-OSS-120B	0.1433	$-50.5$ %	0.1740	0.1304
Historical Baseline	0.0952	0%	—	—

Table 2: Model performance on the held-out test set. Lower Brier score and ECE indicate better performance, while higher Brier skill score and precision are better.

Figure 1: Aggregate performance on the held-out test set

Figure 2: Reliability diagram on the test set showing empirical disruption rates as a function of predicted disruption probabilities.

Dimension	Pretrained Model	Fine-tuned Model
Structure	Short, largely heuristic reasoning	Multi-step reasoning with explicit intermediate structure
Temporal orientation	Describes recent events	Connects current signals to future outcomes
Quantitative grounding	Rare use of numerical reasoning	Frequent use of thresholds, distributions, and rough calculations
Base rate usage	Implicit or absent	Explicit anchoring to baseline probabilities
Uncertainty handling	Single-pass conclusions	Iterative refinement with intermediate estimates
Use of news	Summarizes salient events	Selectively incorporates signals relevant to disruption risk

Table 3: Systematic differences in reasoning behavior before and after training.

Behavior	Pretrained	Fine-tuned
Base rate anchoring	0.09	0.50
Statistical modeling	0.48	1.00
Explicit forecasting model	0.25	0.96
Evidence $\rightarrow$ outcome linkage	0.67	0.70
Probabilistic synthesis	0.94	1.00
Uncertainty refinement	0.33	1.00

Table 4: Frequency of reasoning behaviors detected by automated evaluator. Values represent the fraction of responses exhibiting each behavior.

Explore similar work

CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference
Sep 29, 2026Wenjin Liu, Chenxi Wang, Yue Lu +2Probabilistic Forecasting Language Model Calibration
Hithink research · Nanyang Technological University
What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
Jul 9, 2026Raphaël Sarfati, Pratyush Ranjan Tiwari, Siddharth Boppana +3CoT Faithfulness Language Model Calibration
goodfire · eternis
Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning
Sep 2, 2026Christos Petridis, Zoran Obradovic, Mladen KezunovicLLM Evaluation Zero-Shot Learning
Temple University · Texas A&M University

Sep 29, 2026J/K move · Enter open · S save

CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference

Wenjin Liu, Chenxi Wang, Yue Lu, Zhe Cui+1

Hithink research · Nanyang Technological University

Probabilistic Forecasting Language Model Calibration Event Time Prediction Causal Reasoning in Language Models

Open paper AskPDF arXiv

Large language models have achieved significant progress in event forecasting, yet their probability outputs exhibit systematic calibration bias that varies heterogeneously across different domains and question types, undermining the trustworthiness of probabilistic outputs for decision-making under uncertainty. However, existing calibration methods typically correct probability outputs after prediction is complete, without modeling the structural sources of bias within the prediction process itself. To address this challenge, we decompose probabilistic prediction over causal-temporal hypergraphs into three stages, evidence weighting, evidence aggregation, and source fusion, and propose CHAIN, which designs stage-specific mechanisms to mitigate bias at each stage: (i) modulating the temporal decay function by causal topological distance, (ii) aggregating approximately independent causal chains via Noisy-OR after direction-aware deduplication, and (iii) driving adaptive fusion by causal coverage and directional balance. Experimental results on cross-domain forecasting benchmarks show CHAIN outperforms existing methods in expected calibration error, Brier score, and accuracy. Our project is available at https://github.com/QwenQKing/Chain.

Abstract

Explore similar work

CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning