Large language models have achieved significant progress in event forecasting, yet their probability outputs exhibit systematic calibration bias that varies heterogeneously across different domains and question types, undermining the trustworthiness of probabilistic outputs for decision-making under uncertainty. However, existing calibration methods typically correct probability outputs after prediction is complete, without modeling the structural sources of bias within the prediction process itself. To address this challenge, we decompose probabilistic prediction over causal-temporal hypergraphs into three stages, evidence weighting, evidence aggregation, and source fusion, and propose CHAIN, which designs stage-specific mechanisms to mitigate bias at each stage: (i) modulating the temporal decay function by causal topological distance, (ii) aggregating approximately independent causal chains via Noisy-OR after direction-aware deduplication, and (iii) driving adaptive fusion by causal coverage and directional balance. Experimental results on cross-domain forecasting benchmarks show CHAIN outperforms existing methods in expected calibration error, Brier score, and accuracy. Our project is available at https://github.com/QwenQKing/Chain.
Figures & tables
Figure 1: An illustration of the proposed Chain .
Figure 2: Inference-time approaches to calibrated event forecasting: existing methods and Chain .
Figure 3: The overall framework of Chain , with CTVF, Noisy-OR aggregation, and adaptive fusion.
Method
AI-Futures
Metaculus
Polymarket
Future-as-Label
Average
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
GPT-4o-mini
LLM Direct
15.00
25.69
59.38
22.27
28.79
52.34
20.04
23.19
70.31
21.29
29.97
53.91
19.65
26.91
58.98
LLM CoT
12.89
20.27
71.09
14.22
23.58
64.06
17.42
24.88
64.06
22.62
29.74
51.56
16.79
24.62
62.70
AutoCast
32.85
32.49
43.75
37.77
37.40
42.19
22.38
25.56
68.75
34.92
35.30
49.22
31.98
32.69
50.98
NaiveRAG
26.99
28.79
43.75
28.71
30.72
46.88
14.18
23.96
68.75
36.64
36.62
40.62
26.63
30.02
50.00
Table 1: Main results of Chain and the baselines across four in-distribution forecasting datasets, which are reported on three LLMs with best in bold. All values are in %.
Method
CTO
FOReCAst
Golf-Forecasting
KalshiBench
Average
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
GPT-4o-mini
LLM Direct
18.83
27.61
55.47
12.27
23.43
67.19
19.77
27.18
53.91
20.04
27.81
59.38
17.72
26.51
58.98
LLM CoT
18.12
26.88
56.25
16.37
24.78
60.16
11.25
25.13
60.16
16.51
26.71
57.81
15.56
25.88
58.59
AutoCast
42.50
43.07
42.19
26.09
28.93
59.38
45.35
43.72
47.66
44.30
44.26
41.41
39.56
39.99
47.66
NaiveRAG
30.04
32.62
42.97
30.35
33.02
50.00
24.14
30.31
48.44
30.94
34.34
44.53
28.87
32.57
46.48
Table 2: Performance comparison of Chain and the baselines on 4 OOD datasets across 2 LLMs.
Figure 4: Calibration and predictive performance of Chain for four OOD datasets on DeepSeek-V3.
Method
AI-Futures
Metaculus
Polymarket
Future-as-Label
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
w/o CTVF
11.68
19.43
73.44
10.65
19.59
72.66
19.73
17.54
75.00
10.91
23.15
63.28
w/o Noisy-OR
12.79
19.37
75.78
12.61
19.72
71.09
16.22
17.86
78.91
11.03
19.78
71.88
w/o Adaptive α
11.25
17.66
75.78
10.40
18.27
72.66
14.39
17.66
77.34
10.16
19.68
71.88
CHAIN (Full)
8.54
17.34
76.56
9.72
18.05
73.44
13.74
17.41
79.69
10.09
18.82
72.66
Method
CTO
FOReCAst
Golf-Forecasting
KalshiBench
Table 3: Ablation study of Chain , where w/o means without the corresponding component.
Figure 5: Comparison of Chain and post-hoc calibration methods on DeepSeek-V3 (lower is better ).
Figure 6: DeepSeek-V3 macro-average ECE/Brier over eight ID/OOD datasets: adaptive vs. fixed α .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Hypergraph and Causal Extraction Prompt.
Table 4: Comparison of parameter settings across the three baseline categories and Chain .
Symbol
Meaning
Value
Sensitivity
dmax
reachability/search depth cap
4
swept {2,3,4,5}
θ
Jaccard threshold for same-polarity dedup
0.5
swept {0.3,0.5,0.7}
η
recurrence saturation temperature in Vsal
5.0
—
Mr(e)
default when stored similarity is unavailable
0.5
—
vfb
cutoff-safe missing-time CTVF fallback
0.25
—
ksat
reliability saturation anchor
10
—
Appendix
Table 5: Core Chain -specific hyperparameters and default values used unless otherwise specified.
Backbone
ΔECE↓
ΔBrier↓
ΔAcc↑
GPT-4o-mini
−4.77
−5.57
+9.57
Gemini-2.0-Flash
−4.15
−4.34
+8.20
DeepSeek-V3
−3.07
−3.30
+7.42
Appendix
Table 6: Chain ’s gain over the strongest baseline by backbone LLM. For each backbone and each metric, the strongest baseline is identified by its mean over the four in-distribution datasets, and the reported value is the gap between Chain ’s mean and this baseline’s mean. The hypergraph is extracted with GPT-4o-mini in all rows.
Method
CTO
FOReCAst
Golf-Forecasting
KalshiBench
Average
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
ECE ↓
Brier ↓
Acc ↑
LLM Direct
15.47
25.80
60.16
9.72
20.03
72.66
14.77
26.38
57.03
16.09
25.30
60.94
14.01
24.38
62.70
LLM CoT
16.52
25.01
57.81
10.80
20.28
71.09
17.76
29.48
54.69
20.16
25.54
58.59
16.31
25.08
60.55
AutoCast
12.50
24.27
61.72
16.33
25.61
57.81
13.32
27.53
56.25
13.24
26.28
53.91
13.85
25.92
57.42
NaiveRAG
19.84
25.79
60.16
19.56
27.31
60.16
11.13
26.63
49.22
15.59
28.68
46.88
16.53
27.10
54.10
GraphRAG
13.28
25.29
59.38
8.55
20.77
71.09
14.22
26.61
53.91
18.74
26.46
53.12
13.70
24.78
59.38
Appendix
Table 7: Performance comparison of Chain and the baselines on 4 OOD datasets with DeepSeek-V3.
Parameter
Evaluated settings
Default
Chain budget Bπ
50, 100, 200, 400
200
Maximum search depth dmax
2, 3, 4, 5
4
Fan-out cap Fmax
2, 4, 8, 16, unlimited
unlimited
Pruning threshold τΦ
0, 0.001, 0.005, 0.02, 0.05
0.005
Appendix
Table 8: Evaluated search-budget, depth, fan-out, and pruning-threshold settings, including the defaults used in the main experiments.
Parameter
Setting
ECE ↓
Brier ↓
Accuracy ↑
Chain budget Bπ
50
6.43
20.13
67.58
100
6.62
20.67
67.58
200†
6.85
20.70
67.58
400
6.68
20.70
67.58
Maximum search depth dmax
2
6.53
20.14
67.97
3
5.90
20.47
67.58
Appendix
Table 9: Predictive performance under different search configurations.
τΦ
Pre-dedup.
Post-dedup.
Mean time (s)
Relative time ↓
0
30.76
13.31
10.48
1.00
0.005†
30.76
13.31
2.47
0.24
0.02
29.68
13.15
1.70
0.16
0.05
24.16
12.32
1.25
0.12
Appendix
Table 10: Candidate-chain counts and computational cost under different pruning thresholds. Relative runtime is normalized by τΦ=0 .
Source-record fraction
ECE ↓
Brier ↓
Accuracy ↑
25%
8.62
22.24
63.87
50%
9.24
21.87
64.45
75%
8.86
21.58
65.23
100%
6.85
20.70
67.58
Appendix
Table 11: Predictive performance under different causal-temporal graph scales. Graph scale denotes the configured source-record sampling fraction; results are macro-averaged over four datasets.
Dataset
Hypergraph extractor
ECE ↓
Brier ↓
Accuracy ↑
AI-Futures
GPT-4o-mini
8.54
17.34
76.56
AI-Futures
GPT-4.1-nano
10.38
17.94
73.44
Metaculus
GPT-4o-mini
9.72
18.05
73.44
Metaculus
GPT-4.1-nano
12.26
19.40
74.22
Polymarket
GPT-4o-mini
13.74
17.41
79.69
Polymarket
GPT-4.1-nano
11.21
21.49
67.97
Appendix
Table 12: Performance of Chain with different hypergraph extractors across eight datasets.
Method
Extractor
ECE ↓
Brier ↓
Accuracy ↑
Direct-MO
None
15.21
23.47
60.25
Chain
GPT-4.1-nano
9.89
21.10
67.48
Chain
GPT-4o-mini
8.14
19.89
70.02
Appendix
Table 13: Macro-averaged performance across eight datasets under different hypergraph-extractor settings.
Method
ECE ↓
Brier ↓
NLL ↓
Acc ↑
Temperature Scaling
12.81
23.72
0.6714
61.49
Platt Scaling
20.21
26.69
0.7405
54.43
Isotonic Regression
20.27
26.73
0.7860
52.90
Beta Calibration
20.61
26.64
0.7389
53.12
Venn–Abers
20.04
26.60
0.7384
52.86
Binary Dirichlet
20.61
26.64
0.7389
53.12
Appendix
Table 14: Comparison with modern post-hoc calibration methods under a leakage-safe fitting protocol.
Method
LLM calls/case
Tokens/case
ECE ↓
Brier ↓
NLL ↓
Acc ↑
Direct-MO
1.00
550
15.21
23.47
0.6627
60.25
Five-sample Direct-MO
5.00
2,751
14.91
23.07
0.6514
60.84
Chain
5.00
13,371
8.14
19.89
0.5835
70.02
Appendix
Table 15: LLM forecasting-call control and multi-sample Direct-MO comparison with DeepSeek-V3 across eight datasets. ECE, Brier, and Accuracy are in %.
Method
Adaptive ECE-10 ↓
Debiased RMSCE ↓
Plugin RMSCE ↓
Jackknife ECE ↓
LLM Direct
19.144
16.654
20.639
15.606
Chain
11.742
4.037
11.474
4.312
Appendix
Table 16: Macro-averaged calibration errors under four estimators across 24 evaluated dataset–backbone combinations spanning 8 datasets and 3 backbone language models.
Method
MCE-10 ↓
Reliability-10 ↓
Resolution-10 ↑
Brier ↓
NLL ↓
Acc. ↑
LLM Direct
39.121
4.530
2.805
25.314
0.721
61.491
Chain
22.701
1.428
4.716
20.178
0.590
69.466
Appendix
Table 17: Extended calibration diagnostics and predictive performance, macro-averaged over 24 dataset–backbone combinations. All metrics except NLL are reported as their original values multiplied by 100.
Causal coverage Ω(q)
Record share (%)
Mean α(q)
Median α(q)
Ω(q)=0
49.22
0.000
0.000
0<Ω(q)<0.10
22.46
0.309
0.305
0.10≤Ω(q)<0.25
7.42
0.399
0.404
0.25≤Ω(q)<0.50
3.22
0.541
0.536
0.50≤Ω(q)≤1.00
17.68
0.600
0.600
Appendix
Table 18: Adaptive fusion weights stratified by causal coverage on DeepSeek-V3 across the evaluation sets of all eight datasets.
Component
Evidence-occurrence weighted
Case-balanced
Analysis unit
Each occurrence of a non-target evidence node in a retained chain
Each case–distance stratum
Distance strata
d(n,q)∈{1,…,dmax}
d(n,q)∈{1,…,dmax}
Aggregation
Mean VCTVF(n,q) across node occurrences at each distance
Mean within each case and distance, followed by an average across cases
Target handling
Exclude n∈T(q)
Exclude n∈T(q)
Appendix
Table 19: Definition of the node-level causal-distance evidence audit.
Perturbation
Δ ECE ↓
Δ Brier ↓
Δ Accuracy ↑
Δ NLL ↓
Unperturbed Chain
0.000
0.000
0.000
0.00000
Shuffle edge strengths
−0.034
+0.024
−0.098
+0.00033
Complement strengths ( w←1−w )
+0.829
+0.015
−0.195
−0.00003
Reverse causal-edge directions
+1.689
+1.996
−5.371
+0.06869
Appendix
Table 20: Macro-average performance changes across eight datasets under causal-edge direction and strength-order perturbations with DeepSeek-V3, relative to the unperturbed configuration. ECE, Brier, and Accuracy changes are in percentage points.
Dataset
Common cutoff
Documents
Chunks
Test-record exclusion
Shared boundary
Clinical Trial Outcomes
2024-12-20
11
24
Yes
Identical
FOReCAst
2023-01-01
9
23
Yes
Identical
Golf-Forecasting
2025-10-12
9
27
Yes
Identical
KalshiBench
2025-08-19
12
39
Yes
Identical
Appendix
Table 21: Corpus-boundary audit for the domain-specific retrieval-corpus control.
Method
ECE ↓
Brier ↓
NLL ↓
Acc ↑
NaiveRAG
12.71
23.45
0.663
60.94
GraphRAG
17.57
24.15
0.705
59.57
HyperGraphRAG
12.19
22.95
0.650
62.70
Chain
8.22
22.57
0.640
63.09
Appendix
Table 22: Macro-averaged performance under the domain-specific retrieval-corpus control over four OOD datasets with DeepSeek-V3.
Figure 11: Single-question forecast comparison for a resolved forecasting question, comparing LLM Direct, five RAG methods (NaiveRAG, GraphRAG, LightRAG, HippoRAG, HyperGraphRAG), and five post-hoc calibrators (Temperature Scaling, Platt Scaling, Isotonic Regression, Histogram Binning, Conformal-Score Probability Adjustment) based on DeepSeek-V3, with Chain (Ours). The displayed 100∣p−y∣ is the absolute probability error for this instance, not dataset-level ECE.
Prediction source
P(yes)
∣P−P⋆∣
LLM Direct (external reference)
0.3500
0.6500
Pllm (LLM inside Chain )
0.6000
0.4000
Pcausal (Noisy-OR over 39 chains)
0.99999
0.00001
p^ (Adaptive Fusion, α=0.60 )
0.8400
0.1600
Appendix
Table 23: External reference prediction and internal probability components of Chain in the case study (ground truth: yes , P⋆=1 ). The first row provides the standalone LLM Direct reference; the remaining rows report Chain ’s LLM-side estimate, causal estimate, and final adaptive-fusion output.
Metric
Median Δ
Wilcoxon p
Paired- tp
Wins
Binomial p
ΔECE↓
−0.581
5.96×10−8
5.46×10−4
24/24
5.96×10−8
ΔBrier↓
−1.765
5.96×10−8
7.37×10−7
24/24
5.96×10−8
ΔAcc↑
+2.734
5.96×10−8
4.34×10−7
24/24
5.96×10−8
Appendix
Table 24: Configuration-level paired comparisons of Chain with the metric-specific strongest baseline across 24 (dataset, backbone) configurations spanning three backbone LLMs and eight forecasting datasets. All tests are one-sided with the alternative hypothesis “ Chain better” ( ΔECE,ΔBrier<0 ; ΔAcc>0 ); n=24 . Median differences are in percentage points.
Metric
Favorable comparisons
Median Δ (pp)
Favorable 95% CI
Raw p<.05
Holm p<.05
ECE ↓
120/120
−9.2591
49/120
89/120
44/120
Brier ↓
120/120
−5.7738
75/120
84/120
27/120
Accuracy ↑
120/120
+12.1094
58/120
67/120
20/120
Appendix
Table 25: Question-level paired comparisons between Chain and five retrieval-augmented baselines across three backbone LLMs and eight datasets. Δ denotes Chain minus the baseline in percentage points; the CI column counts paired comparisons whose 95% bootstrap interval lies entirely in the favorable direction.
Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood p∼Beta(α,β) and outcome y∼Bernoulli(p), with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.
Hui Dai, Ryan Teehan, Parsa Torabian +1
Agentic Learning AI Lab, New York University · Chronologies AI
Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.
Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether agents genuinely forecast requires more than final-answer accuracy: a model may be correct by recalling memorized training facts, citing fabricated evidence, or producing an unsupported causal story. We present WorldReasoner, an evaluation framework for temporally valid event forecasting. Each task gives an agent a resolved forecasting question, a simulated forecast date, and access only to evidence available before that date; after resolution, the framework scores the submitted probability, cited evidence, and optional causal event graph. WorldReasoner reports three complementary axes: outcome quality against resolved answers, evidence quality over cited sources, and reasoning quality against post-resolution hindsight graphs. The benchmark is built by an agentic construction pipeline that generates forecasting questions, collects time-stamped evidence, and builds hindsight reference graphs at scale, yielding 345 resolved tasks derived from 14,141 articles with graphs covering 8,087 extracted events. Across six controlled agent settings, temporally valid retrieval is the strongest driver of outcome accuracy; causal graph construction improves key-event recovery; and correct graph-enabled forecasts are more strongly grounded in key events and relevant sources, yet agents still struggle to convert grounded evidence into calibrated probabilities.
Yizhou Chi, Eric Chamoun, Zifeng Ding +1
Department of Computer Science and Technology University of Cambridge