CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference

Wenjin Liu; Chenxi Wang; Yue Lu; Zhe Cui; Haoran Luo

cs.LGSep 29, 2026

CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference

Authors: Wenjin Liu, Chenxi Wang, Yue Lu, Zhe Cui, Haoran Luo

Organizations: Hithink research · Nanyang Technological University

Abstract

Large language models have achieved significant progress in event forecasting, yet their probability outputs exhibit systematic calibration bias that varies heterogeneously across different domains and question types, undermining the trustworthiness of probabilistic outputs for decision-making under uncertainty. However, existing calibration methods typically correct probability outputs after prediction is complete, without modeling the structural sources of bias within the prediction process itself. To address this challenge, we decompose probabilistic prediction over causal-temporal hypergraphs into three stages, evidence weighting, evidence aggregation, and source fusion, and propose CHAIN, which designs stage-specific mechanisms to mitigate bias at each stage: (i) modulating the temporal decay function by causal topological distance, (ii) aggregating approximately independent causal chains via Noisy-OR after direction-aware deduplication, and (iii) driving adaptive fusion by causal coverage and directional balance. Experimental results on cross-domain forecasting benchmarks show CHAIN outperforms existing methods in expected calibration error, Brier score, and accuracy. Our project is available at https://github.com/QwenQKing/Chain.

Figures & tables

Figure 1: An illustration of the proposed Chain .

Figure 2: Inference-time approaches to calibrated event forecasting: existing methods and Chain .

Figure 3: The overall framework of Chain , with CTVF, Noisy-OR aggregation, and adaptive fusion.

Method	AI-Futures			Metaculus			Polymarket			Future-as-Label			Average
Method	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$
GPT-4o-mini
LLM Direct	15.00	25.69	59.38	22.27	28.79	52.34	20.04	23.19	70.31	21.29	29.97	53.91	19.65	26.91	58.98
LLM CoT	12.89	20.27	71.09	14.22	23.58	64.06	17.42	24.88	64.06	22.62	29.74	51.56	16.79	24.62	62.70
AutoCast	32.85	32.49	43.75	37.77	37.40	42.19	22.38	25.56	68.75	34.92	35.30	49.22	31.98	32.69	50.98
NaiveRAG	26.99	28.79	43.75	28.71	30.72	46.88	14.18	23.96	68.75	36.64	36.62	40.62	26.63	30.02	50.00

Table 1: Main results of Chain and the baselines across four in-distribution forecasting datasets, which are reported on three LLMs with best in bold. All values are in %.

Method	CTO			FOReCAst			Golf-Forecasting			KalshiBench			Average
Method	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$
GPT-4o-mini
LLM Direct	18.83	27.61	55.47	12.27	23.43	67.19	19.77	27.18	53.91	20.04	27.81	59.38	17.72	26.51	58.98
LLM CoT	18.12	26.88	56.25	16.37	24.78	60.16	11.25	25.13	60.16	16.51	26.71	57.81	15.56	25.88	58.59
AutoCast	42.50	43.07	42.19	26.09	28.93	59.38	45.35	43.72	47.66	44.30	44.26	41.41	39.56	39.99	47.66
NaiveRAG	30.04	32.62	42.97	30.35	33.02	50.00	24.14	30.31	48.44	30.94	34.34	44.53	28.87	32.57	46.48

Table 2: Performance comparison of Chain and the baselines on 4 OOD datasets across 2 LLMs.

Figure 4: Calibration and predictive performance of Chain for four OOD datasets on DeepSeek-V3.

Method	AI-Futures			Metaculus			Polymarket			Future-as-Label
Method	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$
w/o CTVF	11.68	19.43	73.44	10.65	19.59	72.66	19.73	17.54	75.00	10.91	23.15	63.28
w/o Noisy-OR	12.79	19.37	75.78	12.61	19.72	71.09	16.22	17.86	78.91	11.03	19.78	71.88
w/o Adaptive $\alpha$	11.25	17.66	75.78	10.40	18.27	72.66	14.39	17.66	77.34	10.16	19.68	71.88
CHAIN (Full)	8.54	17.34	76.56	9.72	18.05	73.44	13.74	17.41	79.69	10.09	18.82	72.66
Method	CTO			FOReCAst			Golf-Forecasting			KalshiBench

Table 3: Ablation study of Chain , where w/o means without the corresponding component.

Panel of Figure 5 — Figure 5: Comparison of Chain and post-hoc calibration methods on DeepSeek-V3 (lower is better ).

Figure 6: DeepSeek-V3 macro-average ECE/Brier over eight ID/OOD datasets: adaptive vs. fixed

\alpha

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Hypergraph and Causal Extraction Prompt.

Figure 8: Reasoning Direction Generation Prompt.

Figure 9: Multi-Outcome Probabilistic Reasoning Prompt.

Figure 10: Outcome-to-Answer Mapping Prompt.

Group	Parameter	Direct	RAG	Post-hoc	Chain
Models	Backbone LLMs	3 LLMs	3 LLMs	3 LLMs	3 LLMs
	Construction LLM	–	GPT-4o-mini	–	GPT-4o-mini
	Encoder	–	text-embedding-3-small	–	text-embedding-3-small
	Embedding dim	–	1536	–	1536
Inference	Temperature	0.3	0.3	0.3	0.3
	Max tokens	8192	8192	8192	8192

Appendix

Table 4: Comparison of parameter settings across the three baseline categories and Chain .

Symbol	Meaning	Value	Sensitivity
$d_{\max}$	reachability/search depth cap	$4$	swept $\{2,3,4,5\}$
$\theta$	Jaccard threshold for same-polarity dedup	$0.5$	swept $\{0.3,0.5,0.7\}$
$\eta$	recurrence saturation temperature in $V_{\mathrm{sal}}$	$5.0$	—
$M_{r}(e)$	default when stored similarity is unavailable	$0.5$	—
$v_{\mathrm{fb}}$	cutoff-safe missing-time CTVF fallback	$0.25$	—
$k_{\mathrm{sat}}$	reliability saturation anchor	$10$	—

Appendix

Table 5: Core Chain -specific hyperparameters and default values used unless otherwise specified.

Backbone	$\bm{\Delta\text{ECE}\downarrow}$	$\bm{\Delta\text{Brier}\downarrow}$	$\bm{\Delta\text{Acc}\uparrow}$
GPT-4o-mini	$\mathbf{-4.77}$	$\mathbf{-5.57}$	$\mathbf{+9.57}$
Gemini-2.0-Flash	$-4.15$	$-4.34$	$+8.20$
DeepSeek-V3	$-3.07$	$-3.30$	$+7.42$

Appendix

Table 6: Chain ’s gain over the strongest baseline by backbone LLM. For each backbone and each metric, the strongest baseline is identified by its mean over the four in-distribution datasets, and the reported value is the gap between Chain ’s mean and this baseline’s mean. The hypergraph is extracted with GPT-4o-mini in all rows.

Method	CTO			FOReCAst			Golf-Forecasting			KalshiBench			Average
Method	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$
LLM Direct	15.47	25.80	60.16	9.72	20.03	72.66	14.77	26.38	57.03	16.09	25.30	60.94	14.01	24.38	62.70
LLM CoT	16.52	25.01	57.81	10.80	20.28	71.09	17.76	29.48	54.69	20.16	25.54	58.59	16.31	25.08	60.55
AutoCast	12.50	24.27	61.72	16.33	25.61	57.81	13.32	27.53	56.25	13.24	26.28	53.91	13.85	25.92	57.42
NaiveRAG	19.84	25.79	60.16	19.56	27.31	60.16	11.13	26.63	49.22	15.59	28.68	46.88	16.53	27.10	54.10
GraphRAG	13.28	25.29	59.38	8.55	20.77	71.09	14.22	26.61	53.91	18.74	26.46	53.12	13.70	24.78	59.38

Appendix

Table 7: Performance comparison of Chain and the baselines on 4 OOD datasets with DeepSeek-V3.

Parameter	Evaluated settings	Default
Chain budget $B_{\pi}$	50, 100, 200, 400	200
Maximum search depth $d_{\max}$	2, 3, 4, 5	4
Fan-out cap $F_{\max}$	2, 4, 8, 16, unlimited	unlimited
Pruning threshold $\tau_{\Phi}$	0, 0.001, 0.005, 0.02, 0.05	0.005

Appendix

Table 8: Evaluated search-budget, depth, fan-out, and pruning-threshold settings, including the defaults used in the main experiments.

Parameter	Setting	ECE $\downarrow$	Brier $\downarrow$	Accuracy $\uparrow$
Chain budget $B_{\pi}$	50	6.43	20.13	67.58
	100	6.62	20.67	67.58
	$200^{\dagger}$	6.85	20.70	67.58
	400	6.68	20.70	67.58
Maximum search depth $d_{\max}$	2	6.53	20.14	67.97
	3	5.90	20.47	67.58

Appendix

Table 9: Predictive performance under different search configurations.

$\bm{\tau_{\Phi}}$	Pre-dedup.	Post-dedup.	Mean time (s)	Relative time $\downarrow$
0	30.76	13.31	10.48	1.00
$0.005^{\dagger}$	30.76	13.31	2.47	0.24
0.02	29.68	13.15	1.70	0.16
0.05	24.16	12.32	1.25	0.12

Appendix

Table 10: Candidate-chain counts and computational cost under different pruning thresholds. Relative runtime is normalized by

\tau_{\Phi}=0

Source-record fraction	ECE $\downarrow$	Brier $\downarrow$	Accuracy $\uparrow$
25%	8.62	22.24	63.87
50%	9.24	21.87	64.45
75%	8.86	21.58	65.23
100%	6.85	20.70	67.58

Appendix

Table 11: Predictive performance under different causal-temporal graph scales. Graph scale denotes the configured source-record sampling fraction; results are macro-averaged over four datasets.

Dataset	Hypergraph extractor	ECE $\downarrow$	Brier $\downarrow$	Accuracy $\uparrow$
AI-Futures	GPT-4o-mini	8.54	17.34	76.56
AI-Futures	GPT-4.1-nano	10.38	17.94	73.44
Metaculus	GPT-4o-mini	9.72	18.05	73.44
Metaculus	GPT-4.1-nano	12.26	19.40	74.22
Polymarket	GPT-4o-mini	13.74	17.41	79.69
Polymarket	GPT-4.1-nano	11.21	21.49	67.97

Appendix

Table 12: Performance of Chain with different hypergraph extractors across eight datasets.

Method	Extractor	ECE $\downarrow$	Brier $\downarrow$	Accuracy $\uparrow$
Direct-MO	None	15.21	23.47	60.25
Chain	GPT-4.1-nano	9.89	21.10	67.48
Chain	GPT-4o-mini	8.14	19.89	70.02

Appendix

Table 13: Macro-averaged performance across eight datasets under different hypergraph-extractor settings.

Method	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Acc $\uparrow$
Temperature Scaling	12.81	23.72	0.6714	61.49
Platt Scaling	20.21	26.69	0.7405	54.43
Isotonic Regression	20.27	26.73	0.7860	52.90
Beta Calibration	20.61	26.64	0.7389	53.12
Venn–Abers	20.04	26.60	0.7384	52.86
Binary Dirichlet	20.61	26.64	0.7389	53.12

Appendix

Table 14: Comparison with modern post-hoc calibration methods under a leakage-safe fitting protocol.

Method	LLM calls/case	Tokens/case	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Acc $\uparrow$
Direct-MO	1.00	550	15.21	23.47	0.6627	60.25
Five-sample Direct-MO	5.00	2,751	14.91	23.07	0.6514	60.84
Chain	5.00	13,371	8.14	19.89	0.5835	70.02

Appendix

Table 15: LLM forecasting-call control and multi-sample Direct-MO comparison with DeepSeek-V3 across eight datasets. ECE, Brier, and Accuracy are in %.

Method	Adaptive ECE-10 $\downarrow$	Debiased RMSCE $\downarrow$	Plugin RMSCE $\downarrow$	Jackknife ECE $\downarrow$
LLM Direct	19.144	16.654	20.639	15.606
Chain	11.742	4.037	11.474	4.312

Appendix

Table 16: Macro-averaged calibration errors under four estimators across 24 evaluated dataset–backbone combinations spanning 8 datasets and 3 backbone language models.

Method	MCE-10 $\downarrow$	Reliability-10 $\downarrow$	Resolution-10 $\uparrow$	Brier $\downarrow$	NLL $\downarrow$	Acc. $\uparrow$
LLM Direct	39.121	4.530	2.805	25.314	0.721	61.491
Chain	22.701	1.428	4.716	20.178	0.590	69.466

Appendix

Table 17: Extended calibration diagnostics and predictive performance, macro-averaged over 24 dataset–backbone combinations. All metrics except NLL are reported as their original values multiplied by 100.

Causal coverage $\Omega(q)$	Record share (%)	Mean $\alpha(q)$	Median $\alpha(q)$
$\Omega(q)=0$	49.22	0.000	0.000
$0<\Omega(q)<0.10$	22.46	0.309	0.305
$0.10\leq\Omega(q)<0.25$	7.42	0.399	0.404
$0.25\leq\Omega(q)<0.50$	3.22	0.541	0.536
$0.50\leq\Omega(q)\leq 1.00$	17.68	0.600	0.600

Appendix

Table 18: Adaptive fusion weights stratified by causal coverage on DeepSeek-V3 across the evaluation sets of all eight datasets.

Component	Evidence-occurrence weighted	Case-balanced
Analysis unit	Each occurrence of a non-target evidence node in a retained chain	Each case–distance stratum
Distance strata	$d(n,q)\in\{1,\ldots,d_{\max}\}$	$d(n,q)\in\{1,\ldots,d_{\max}\}$
Aggregation	Mean $V_{\mathrm{CTVF}}(n,q)$ across node occurrences at each distance	Mean within each case and distance, followed by an average across cases
Target handling	Exclude $n\in\mathcal{T}(q)$	Exclude $n\in\mathcal{T}(q)$

Appendix

Table 19: Definition of the node-level causal-distance evidence audit.

Perturbation	$\Delta$ ECE $\downarrow$	$\Delta$ Brier $\downarrow$	$\Delta$ Accuracy $\uparrow$	$\Delta$ NLL $\downarrow$
Unperturbed Chain	0.000	0.000	0.000	0.00000
Shuffle edge strengths	$-0.034$	$+0.024$	$-0.098$	$+0.00033$
Complement strengths ( $w\leftarrow 1-w$ )	$+0.829$	$+0.015$	$-0.195$	$-0.00003$
Reverse causal-edge directions	$+1.689$	$+1.996$	$-5.371$	$+0.06869$

Appendix

Table 20: Macro-average performance changes across eight datasets under causal-edge direction and strength-order perturbations with DeepSeek-V3, relative to the unperturbed configuration. ECE, Brier, and Accuracy changes are in percentage points.

Dataset	Common cutoff	Documents	Chunks	Test-record exclusion	Shared boundary
Clinical Trial Outcomes	2024-12-20	11	24	Yes	Identical
FOReCAst	2023-01-01	9	23	Yes	Identical
Golf-Forecasting	2025-10-12	9	27	Yes	Identical
KalshiBench	2025-08-19	12	39	Yes	Identical

Appendix

Table 21: Corpus-boundary audit for the domain-specific retrieval-corpus control.

Method	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Acc $\uparrow$
NaiveRAG	12.71	23.45	0.663	60.94
GraphRAG	17.57	24.15	0.705	59.57
HyperGraphRAG	12.19	22.95	0.650	62.70
Chain	8.22	22.57	0.640	63.09

Appendix

Table 22: Macro-averaged performance under the domain-specific retrieval-corpus control over four OOD datasets with DeepSeek-V3.

Figure 11: Single-question forecast comparison for a resolved forecasting question, comparing LLM Direct, five RAG methods (NaiveRAG, GraphRAG, LightRAG, HippoRAG, HyperGraphRAG), and five post-hoc calibrators (Temperature Scaling, Platt Scaling, Isotonic Regression, Histogram Binning, Conformal-Score Probability Adjustment) based on DeepSeek-V3, with Chain (Ours). The displayed

100|p-y|

is the absolute probability error for this instance, not dataset-level ECE.

Prediction source	$\bm{P(\mathrm{yes})}$	$\bm{\|P-P^{\star}\|}$
LLM Direct (external reference)	$0.3500$	$0.6500$
$P_{\mathrm{llm}}$ (LLM inside Chain )	$0.6000$	$0.4000$
$P_{\mathrm{causal}}$ (Noisy-OR over 39 chains)	$0.99999$	$0.00001$
$\hat{p}$ (Adaptive Fusion, $\alpha=0.60$ )	$\mathbf{0.8400}$	$\mathbf{0.1600}$

Appendix

Table 23: External reference prediction and internal probability components of Chain in the case study (ground truth: yes ,

P^{\star}=1

). The first row provides the standalone LLM Direct reference; the remaining rows report Chain ’s LLM-side estimate, causal estimate, and final adaptive-fusion output.

Metric	Median $\Delta$	Wilcoxon $p$	Paired- $t$ $p$	Wins	Binomial $p$
$\Delta\text{ECE}\!\downarrow$	$-0.581$	$\mathbf{5.96\!\times\!10^{-8}}$	$\mathbf{5.46\!\times\!10^{-4}}$	$24/24$	$\mathbf{5.96\!\times\!10^{-8}}$
$\Delta\text{Brier}\!\downarrow$	$-1.765$	$\mathbf{5.96\!\times\!10^{-8}}$	$\mathbf{7.37\!\times\!10^{-7}}$	$24/24$	$\mathbf{5.96\!\times\!10^{-8}}$
$\Delta\text{Acc}\!\uparrow$	$+2.734$	$\mathbf{5.96\!\times\!10^{-8}}$	$\mathbf{4.34\!\times\!10^{-7}}$	$24/24$	$\mathbf{5.96\!\times\!10^{-8}}$

Appendix

Table 24: Configuration-level paired comparisons of Chain with the metric-specific strongest baseline across

24

(dataset, backbone) configurations spanning three backbone LLMs and eight forecasting datasets. All tests are one-sided with the alternative hypothesis “ Chain better” (

\Delta\text{ECE},\Delta\text{Brier}<0

;

\Delta\text{Acc}>0

);

n=24

. Median differences are in percentage points.

Metric	Favorable comparisons	Median $\Delta$ (pp)	Favorable 95% CI	Raw $\bm{p<.05}$	Holm $\bm{p<.05}$
ECE $\downarrow$	120/120	$-9.2591$	49/120	89/120	44/120
Brier $\downarrow$	120/120	$-5.7738$	75/120	84/120	27/120
Accuracy $\uparrow$	120/120	$+12.1094$	58/120	67/120	20/120

Appendix

Table 25: Question-level paired comparisons between Chain and five retrieval-augmented baselines across three backbone LLMs and eight datasets.

\Delta

denotes Chain minus the baseline in percentage points; the CI column counts paired comparisons whose 95% bootstrap interval lies entirely in the favorable direction.

Explore similar work

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
May 26, 2026Hui Dai, Ryan Teehan, Parsa Torabian +1Post-Hoc Calibration Probabilistic Forecasting
Agentic Learning AI Lab, New York University · Chronologies AI
What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
Jul 9, 2026Raphaël Sarfati, Pratyush Ranjan Tiwari, Siddharth Boppana +3CoT Faithfulness Language Model Calibration
goodfire · eternis
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
Jun 10, 2026Yizhou Chi, Eric Chamoun, Zifeng Ding +1Event Time Prediction LLM Agent Evaluation
Department of Computer Science and Technology University of Cambridge

May 26, 2026J/K move · Enter open · S save

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

Hui Dai, Ryan Teehan, Parsa Torabian, Mengye Ren

Agentic Learning AI Lab, New York University · Chronologies AI

Post-Hoc Calibration Probabilistic Forecasting Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood

p \sim \text{Beta}(α, β)

and outcome

y \sim \text{Bernoulli}(p)

, with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.

Abstract

Explore similar work

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning