Agentic forecasting systems based on large language models (LLMs) are now competitive with expert human forecasters, but these systems still suffer from a critical limitation: the inability to produce forecasts that are logically consistent. For example, forecasts for related events often fail to satisfy the axioms of probability, which in turn implies that related forecasts cannot simultaneously be correct and undermines their utility for decision makers. Enforcing consistency is not trivial -- naive approaches (e.g., projection onto the set of consistent forecasts) can degrade accuracy. We argue that the key to overcoming the apparent tradeoff between consistency and accuracy lies in the reasoning which supports each forecast. We finetune a supervisory language model that leverages this reasoning to jointly reconcile inconsistencies while improving the accuracy of related forecasts. On ForecastBench, our method improves the Brier score of a retrieval-augmented GPT-5.1 forecaster by 9.3% while reducing measures of structural inconsistency by 82-90%. We interrogate our method via a comprehensive series of baselines and ablations, and find that no alternative method matches our gains in consistency and accuracy simultaneously.
Figures & tables
Figure 1: Overview of our method, illustrated with a stylized example. Given a target question to forecast, we generate a cluster of related questions with structural relationships to the original. We then run the upstream agentic LLM forecaster on all questions in the cluster, obtaining initial probability forecasts and reasoning traces for each question. Finally, the crux of our method is to jointly reconcile the forecasts within each cluster using the initial probabilities and reasoning traces.
Metric
Raw
Ours (mean ± std)
%Δ (mean ± std)
Brier score
0.151657
0.137519±0.000900
−9.32%±0.59%
Negation squared error
0.046831
0.005194±0.000662
−88.91%±1.41%
Multi-way partition squared error
0.223655
0.021966±0.001937
−90.18%±0.87%
Monotonicity L1 violation
0.018565
0.003246±0.000219
−82.52%±1.18%
Table 1: Our method’s headline results. Raw denotes the upstream forecaster. Ours reports the mean ± sample standard deviation of our method across three seeds. Lower is better for every metric. %Δ reports the relative percentage change of Ours with respect to Raw .
Method
% Δ Brier
% Δ Pooled partition viol.
% Δ Monotonicity viol.
Ours
−9.32%
−86.12%
−82.52%
Ensemble over 10x Runs
−6.57%
−41.54%
−53.45%
James–Stein shrinkage
−0.88%
−2.83%
−23.65%
Platt scaling
+1.68%
+27.80%
+5.31%
Consistent-set projection
−2.79%
−100.0%
−100.0%
Zero-Shot LLM
+3.98%
−100.0%
−100.0%
Table 2: Comparison of our method with baseline methods, showing the relative change ( %Δ ) in Brier score and consistency violations with respect to Raw for each method. Negative values are better, and the best value in each column is in bold.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Synthetic family
Candidate groups
Judge-approved groups
Groups audited
Audit false passes
95% judge false-pass rate
Negation
1,581
1,580
50
0
5.82%
Monotonic ladder
1,587
1,562
50
0
5.82%
Multi-way partition
404
369
50
0
5.82%
Appendix
Table 3: LLM-judge filtering and manual-audit summary for synthetic relationship families in the test set. The final column reports the one-sided 95% confidence bound on the judge’s false-pass rate.
Metric
Raw
Seed 31
Seed 32
Seed 33
Brier score
0.1517
0.1385(−8.65%)
0.1372(−9.56%)
0.1369(−9.76%)
Negation sq. error
0.0468
0.0056(−88.09%)
0.0044(−90.54%)
0.0056(−88.09%)
Partition sq. error
0.2237
0.0201(−91.00%)
0.0240(−89.27%)
0.0218(−90.27%)
Monotonicity L1 viol.
0.0186
0.0035(−81.31%)
0.0030(−83.66%)
0.0032(−82.59%)
Appendix
Table 4: Per-seed breakdown of our main results shown in Table 1 . Raw denotes the upstream forecaster and is shared across seeds; lower is better for every metric. Each column reports the metric value with its relative change from Raw in parentheses (negative is better). Brier score is computed over the resolved original questions, whereas consistency metrics are computed over the corresponding augmented clusters. Each metric is an equal-weighted average over the 243 -question market subset and the 1,344 -question dataset subset.
Metric
Market ( n=243 )
Dataset ( n=1,344 )
Brier score
0.1451→0.1291(−11.08%)
0.1582→0.1460(−7.71%)
Negation squared error
0.0773→0.0078(−89.92%)
0.0164→0.0026(−84.12%)
Partition squared error
0.1198→0.0249(−79.19%)
0.3275→0.0190(−94.20%)
Monotonicity L1 violation
0.0201→0.0047(−76.65%)
0.0170→0.0018(−89.43%)
Appendix
Table 5: Our main results by question source. Each entry reports Raw → Model, with the relative change from Raw in parentheses. Model values are means across the three training seeds. Lower is better for every metric.
Seed
Raw
Ours
Δ
%Δ
Average
0.176466
0.141092
−0.035373
−20.05%
31
0.176466
0.142999
−0.033467
−18.96%
32
0.176466
0.140033
−0.036433
−20.65%
33
0.176466
0.140246
−0.036220
−20.53%
Appendix
Table 6: Brier score over the complement questions in our test set, computed as an equal-weighted average over the 243 -question market subset and 1,344 -question dataset subset. Raw denotes the upstream forecaster and is shared across seeds. Δ is Model minus Raw, so negative values indicate improvement. The average row is the mean of the per-seed squared losses.
Method
% Δ Brier
% Δ Pooled partition viol.
% Δ Monotonicity viol.
Ours
−9.32%
−86.12%
−82.52%
Ablation: No consistency loss
−9.69%
−51.68%
−37.38%
Ablation: No reasoning traces
−2.67%
−81.72%
−79.37%
Ablation: No cross-question context
−8.88%
−57.75%
−49.78%
Ablation: Frontier GPT-5.1
−5.83%
−18.26%
−2.16%
Appendix
Table 7: The relative improvement of each ablation method with respect to Raw ; negative values indicate better performance. Ours denotes our method, with metrics averaged over three seeds.
Brier
Negation sq. error
Multi-way partition sq. error
Monotonicity L1 violation
0.0012P
0.0002P
0.0002P
0.0002P
Appendix
Table 8: Two-sided paired-bootstrap p-values comparing our method with the raw upstream forecasts on the test set. Bold entries are significant at the 5% level; the superscript P indicates that our method has the lower metric.
Baseline
Brier
Negation sq. error
Multi-way partition sq. error
Monotonicity L1 violation
Ensemble over 10x Runs
0.2088
0.0002P
0.0002P
0.0002P
James–Stein shrinkage
0.0010P
0.0002P
0.0002P
0.0002P
Platt scaling
0.0024P
0.0002P
0.0002P
0.0002P
Consistent-set projection
0.0014P
0.0002B
0.0002B
0.0002B
Zero-Shot GPT-5.1
0.0006P
0.0002B
0.0002B
0.0002B
Appendix
Table 9: Two-sided paired-bootstrap p-values comparing our method with each baseline on the test set. Bold entries are significant at the 5% level; the superscript indicates whether our method (P) or the baseline (B) has the lower metric.
Ablation
Brier
Negation sq. error
Multi-way partition sq. error
Monotonicity L1 violation
No consistency loss
0.7039
0.0002P
0.0002P
0.0002P
No reasoning traces
0.0002P
0.0644P
0.0020P
0.0200P
No cross-question context
0.7763
0.0002P
0.0002P
0.0002P
Frontier GPT-5.1
0.1738
0.0002P
0.0002P
0.0002P
Appendix
Table 10: Two-sided paired-bootstrap p-values comparing our method with each ablation on the test set. Bold entries are significant at the 5% level; the superscript indicates whether our method (P) or the ablation (A) has the lower metric.
Metric
Raw
10-run mean
Δ
Percent Δ
Brier score
0.151657
0.141696
−0.009961
−6.57%
Negation squared error
0.046831
0.011157
−0.035674
−76.18%
Multi-way partition squared error
0.223655
0.176326
−0.047330
−21.16%
Monotonicity L1 violation
0.018565
0.008642
−0.009923
−53.45%
Appendix
Table 11: Arithmetic mean of ten independent GPT-5.1 + Perplexity forecasts on the test set.
Metric
Raw
Classical JS
Δ
Percent Δ
Brier score
0.151657
0.150317
−0.001339
−0.88%
Negation squared error
0.046831
0.039709
−0.007122
−15.21%
Multi-way partition squared error
0.223655
0.220809
−0.002846
−1.27%
Monotonicity L1 violation
0.018565
0.014174
−0.004391
−23.65%
Appendix
Table 12: Classical positive-part James-Stein shrinkage on synthetic-question clusters.
Metric
Raw
Shrunk
Δ
Percent Δ
Brier score
0.151657
0.149164
−0.002492
−1.64%
Negation squared error
0.046831
0.040904
−0.005927
−12.66%
Multi-way partition squared error
0.223655
0.198432
−0.025223
−11.28%
Monotonicity L1 violation
0.018565
0.014030
−0.004535
−24.43%
Appendix
Table 13: Heteroskedastic empirical Bayes shrinkage on synthetic-question clusters.
Metric
Raw
Log-odds Classical JS
Δ
Percent Δ
Brier score
0.151657
0.155155
+0.003498
+2.31%
Negation squared error
0.046831
0.078087
+0.031256
+66.74%
Multi-way partition squared error
0.223655
0.188271
−0.035384
−15.82%
Monotonicity L1 violation
0.018565
0.010500
−0.008065
−43.44%
Appendix
Table 14: Log-odds classical positive-part James-Stein shrinkage on synthetic-question clusters.
Metric
Raw
Zero-target Classical JS
Δ
Percent Δ
Brier score
0.151657
0.149828
−0.001829
−1.21%
Negation squared error
0.046831
0.042051
−0.004779
−10.21%
Multi-way partition squared error
0.223655
0.178662
−0.044993
−20.12%
Monotonicity L1 violation
0.018565
0.017647
−0.000917
−4.94%
Appendix
Table 15: Global classical positive-part James–Stein shrinkage toward the fixed probability target zero.
Metric
Raw
Platt
Δ
Percent Δ
Brier score
0.151657
0.154199
+0.002543
+1.68%
Negation squared error
0.046831
0.075584
+0.028753
+61.40%
Multi-way partition squared error
0.223655
0.212630
−0.011025
−4.93%
Monotonicity L1 violation
0.018565
0.019551
+0.000986
+5.31%
Appendix
Table 16: Fixed-parameter Platt scaling with coefficient 3 .
Metric
Raw
Projection
Δ
Percent Δ
Brier score
0.151657
0.147423
−0.004233
−2.79%
Negation squared error
0.046831
0.000000
−0.046831
−100.00%
Multi-way partition squared error
0.223655
0.000000
−0.223655
−100.00%
Monotonicity L1 violation
0.018565
0.000000
−0.018565
−100.00%
Appendix
Table 17: Projection onto the consistent solution set with synthetic structural constraints enforced exactly.
Metric
Raw
Zero-Shot GPT-5.1
Δ
Percent Δ
Brier score
0.151657
0.157692
+0.006035
+3.98%
Negation squared error
0.046831
0.000000
−0.046831
−100.00%
Multi-way partition squared error
0.223655
0.000000
−0.223655
−100.00%
Monotonicity L1 violation
0.018565
0.000000
−0.018565
−100.00%
Appendix
Table 18: Zero-shot GPT-5.1 joint-reconciliation baseline on synthetic-question clusters.
Metric
Raw
Ablation
Δ
Percent Δ
Brier score
0.151657
0.136959
−0.014697
−9.69%
Negation squared error
0.046831
0.022303
−0.024527
−52.37%
Multi-way partition squared error
0.223655
0.074718
−0.148938
−66.59%
Monotonicity L1 violation
0.018565
0.011626
−0.006939
−37.38%
Appendix
Table 19: Ablation removing the consistency term from the loss on synthetic-question clusters.
Metric
Raw
Seed 31
Seed 32
Seed 33
Brier score
0.1517
0.1366(−9.93%)
0.1368(−9.79%)
0.1375(−9.36%)
Negation sq. error
0.0468
0.0291(−37.81%)
0.0182(−61.16%)
0.0196(−58.15%)
Partition sq. error
0.2237
0.0506(−77.39%)
0.0684(−69.42%)
0.1052(−52.97%)
Monotonicity L1 viol.
0.0186
0.0109(−41.06%)
0.0111(−40.12%)
0.0128(−30.96%)
Appendix
Table 20: Per-seed breakdown of Table 19 . Each column reports the metric value with its relative change from Raw in parentheses (negative is better).
Metric
Raw
Ablation
Δ
Percent Δ
Brier score
0.151657
0.147610
−0.004047
−2.67%
Negation squared error
0.046831
0.007030
−0.039801
−84.99%
Multi-way partition squared error
0.223655
0.029545
−0.194111
−86.79%
Monotonicity L1 violation
0.018565
0.003830
−0.014735
−79.37%
Appendix
Table 21: Ablation removing reasoning traces from prompts on synthetic-question clusters.
Metric
Raw
Seed 31
Seed 32
Seed 33
Brier score
0.1517
0.1473(−2.85%)
0.1476(−2.70%)
0.1479(−2.46%)
Negation sq. error
0.0468
0.0069(−85.16%)
0.0073(−84.50%)
0.0069(−85.30%)
Partition sq. error
0.2237
0.0306(−86.30%)
0.0220(−90.18%)
0.0360(−83.90%)
Monotonicity L1 viol.
0.0186
0.0036(−80.35%)
0.0038(−79.78%)
0.0041(−77.98%)
Appendix
Table 22: Per-seed breakdown of Table 21 . Each column reports the metric value with its relative change from Raw in parentheses (negative is better).
Metric
Raw
Ablation
Δ
Percent Δ
Brier score
0.151657
0.138197
−0.013460
−8.88%
Negation squared error
0.046831
0.013080
−0.033750
−72.07%
Multi-way partition squared error
0.223655
0.095183
−0.128473
−57.44%
Monotonicity L1 violation
0.018565
0.009324
−0.009241
−49.78%
Appendix
Table 23: No cross-question context ablation that uses clusters containing ten independent runs of the same question, as opposed to clusters with related questions.
Metric
Raw
Seed 31
Seed 32
Seed 33
Brier score
0.1517
0.1370(−9.64%)
0.1391(−8.25%)
0.1384(−8.74%)
Negation sq. error
0.0468
0.0147(−68.71%)
0.0126(−73.16%)
0.0120(−74.33%)
Partition sq. error
0.2237
0.0979(−56.21%)
0.0923(−58.71%)
0.0953(−57.41%)
Monotonicity L1 viol.
0.0186
0.0096(−48.28%)
0.0094(−49.28%)
0.0090(−51.77%)
Appendix
Table 24: Per-seed breakdown of Table 23 . Each column reports the metric value with its relative change from Raw in parentheses (negative is better).
Metric
Raw
GPT-5.1
Δ
Percent Δ
Brier score
0.151657
0.142821
−0.008836
−5.83%
Negation squared error
0.046831
0.036621
−0.010210
−21.80%
Multi-way partition squared error
0.223655
0.180908
−0.042748
−19.11%
Monotonicity L1 violation
0.018565
0.018164
−0.000401
−2.16%
Appendix
Table 25: Ablation replacing the finetuned model with GPT-5.1 off-the-shelf.