Logically Consistent Accurate Forecasts
Organizations: Bridgewater AIA Labs · Massachusetts Institute of Technology
Abstract
Agentic forecasting systems based on large language models (LLMs) are now competitive with expert human forecasters, but these systems still suffer from a critical limitation: the inability to produce forecasts that are logically consistent. For example, forecasts for related events often fail to satisfy the axioms of probability, which in turn implies that related forecasts cannot simultaneously be correct and undermines their utility for decision makers. Enforcing consistency is not trivial -- naive approaches (e.g., projection onto the set of consistent forecasts) can degrade accuracy. We argue that the key to overcoming the apparent tradeoff between consistency and accuracy lies in the reasoning which supports each forecast. We finetune a supervisory language model that leverages this reasoning to jointly reconcile inconsistencies while improving the accuracy of related forecasts. On ForecastBench, our method improves the Brier score of a retrieval-augmented GPT-5.1 forecaster by 9.3% while reducing measures of structural inconsistency by 82-90%. We interrogate our method via a comprehensive series of baselines and ablations, and find that no alternative method matches our gains in consistency and accuracy simultaneously.
Figures & tables
| Metric | Raw | Ours (mean std) | (mean std) |
|---|---|---|---|
| Brier score | 0.151657 | ||
| Negation squared error | 0.046831 | ||
| Multi-way partition squared error | 0.223655 | ||
| Monotonicity violation | 0.018565 |
| Method | % Brier | % Pooled partition viol. | % Monotonicity viol. |
|---|---|---|---|
| Ours | |||
| Ensemble over 10x Runs | |||
| James–Stein shrinkage | |||
| Platt scaling | |||
| Consistent-set projection | |||
| Zero-Shot LLM |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Synthetic family | Candidate groups | Judge-approved groups | Groups audited | Audit false passes | judge false-pass rate |
|---|---|---|---|---|---|
| Negation | 1,581 | 1,580 | 50 | 0 | |
| Monotonic ladder | 1,587 | 1,562 | 50 | 0 | |
| Multi-way partition | 404 | 369 | 50 | 0 |
| Metric | Raw | Seed 31 | Seed 32 | Seed 33 |
|---|---|---|---|---|
| Brier score | 0.1517 | |||
| Negation sq. error | 0.0468 | |||
| Partition sq. error | 0.2237 | |||
| Monotonicity viol. | 0.0186 |
| Metric | Market ( ) | Dataset ( ) |
|---|---|---|
| Brier score | ||
| Negation squared error | ||
| Partition squared error | ||
| Monotonicity violation |
| Seed | Raw | Ours | ||
|---|---|---|---|---|
| Average | 0.176466 | 0.141092 | ||
| 31 | 0.176466 | 0.142999 | ||
| 32 | 0.176466 | 0.140033 | ||
| 33 | 0.176466 | 0.140246 |
| Method | % Brier | % Pooled partition viol. | % Monotonicity viol. |
|---|---|---|---|
| Ours | |||
| Ablation: No consistency loss | |||
| Ablation: No reasoning traces | |||
| Ablation: No cross-question context | |||
| Ablation: Frontier GPT-5.1 |
| Brier | Negation sq. error | Multi-way partition sq. error | Monotonicity violation |
|---|---|---|---|
| Baseline | Brier | Negation sq. error | Multi-way partition sq. error | Monotonicity violation |
|---|---|---|---|---|
| Ensemble over 10x Runs | 0.2088 | |||
| James–Stein shrinkage | ||||
| Platt scaling | ||||
| Consistent-set projection | ||||
| Zero-Shot GPT-5.1 |
| Ablation | Brier | Negation sq. error | Multi-way partition sq. error | Monotonicity violation |
|---|---|---|---|---|
| No consistency loss | 0.7039 | |||
| No reasoning traces | ||||
| No cross-question context | 0.7763 | |||
| Frontier GPT-5.1 | 0.1738 |
| Metric | Raw | 10-run mean | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.141696 | ||
| Negation squared error | 0.046831 | 0.011157 | ||
| Multi-way partition squared error | 0.223655 | 0.176326 | ||
| Monotonicity violation | 0.018565 | 0.008642 |
| Metric | Raw | Classical JS | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.150317 | ||
| Negation squared error | 0.046831 | 0.039709 | ||
| Multi-way partition squared error | 0.223655 | 0.220809 | ||
| Monotonicity violation | 0.018565 | 0.014174 |
| Metric | Raw | Shrunk | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.149164 | ||
| Negation squared error | 0.046831 | 0.040904 | ||
| Multi-way partition squared error | 0.223655 | 0.198432 | ||
| Monotonicity violation | 0.018565 | 0.014030 |
| Metric | Raw | Log-odds Classical JS | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.155155 | ||
| Negation squared error | 0.046831 | 0.078087 | ||
| Multi-way partition squared error | 0.223655 | 0.188271 | ||
| Monotonicity violation | 0.018565 | 0.010500 |
| Metric | Raw | Zero-target Classical JS | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.149828 | ||
| Negation squared error | 0.046831 | 0.042051 | ||
| Multi-way partition squared error | 0.223655 | 0.178662 | ||
| Monotonicity violation | 0.018565 | 0.017647 |
| Metric | Raw | Platt | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.154199 | ||
| Negation squared error | 0.046831 | 0.075584 | ||
| Multi-way partition squared error | 0.223655 | 0.212630 | ||
| Monotonicity violation | 0.018565 | 0.019551 |
| Metric | Raw | Projection | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.147423 | ||
| Negation squared error | 0.046831 | 0.000000 | ||
| Multi-way partition squared error | 0.223655 | 0.000000 | ||
| Monotonicity violation | 0.018565 | 0.000000 |
| Metric | Raw | Zero-Shot GPT-5.1 | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.157692 | ||
| Negation squared error | 0.046831 | 0.000000 | ||
| Multi-way partition squared error | 0.223655 | 0.000000 | ||
| Monotonicity violation | 0.018565 | 0.000000 |
| Metric | Raw | Ablation | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.136959 | ||
| Negation squared error | 0.046831 | 0.022303 | ||
| Multi-way partition squared error | 0.223655 | 0.074718 | ||
| Monotonicity violation | 0.018565 | 0.011626 |
| Metric | Raw | Seed 31 | Seed 32 | Seed 33 |
|---|---|---|---|---|
| Brier score | 0.1517 | |||
| Negation sq. error | 0.0468 | |||
| Partition sq. error | 0.2237 | |||
| Monotonicity viol. | 0.0186 |
| Metric | Raw | Ablation | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.147610 | ||
| Negation squared error | 0.046831 | 0.007030 | ||
| Multi-way partition squared error | 0.223655 | 0.029545 | ||
| Monotonicity violation | 0.018565 | 0.003830 |
| Metric | Raw | Seed 31 | Seed 32 | Seed 33 |
|---|---|---|---|---|
| Brier score | 0.1517 | |||
| Negation sq. error | 0.0468 | |||
| Partition sq. error | 0.2237 | |||
| Monotonicity viol. | 0.0186 |
| Metric | Raw | Ablation | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.138197 | ||
| Negation squared error | 0.046831 | 0.013080 | ||
| Multi-way partition squared error | 0.223655 | 0.095183 | ||
| Monotonicity violation | 0.018565 | 0.009324 |
| Metric | Raw | Seed 31 | Seed 32 | Seed 33 |
|---|---|---|---|---|
| Brier score | 0.1517 | |||
| Negation sq. error | 0.0468 | |||
| Partition sq. error | 0.2237 | |||
| Monotonicity viol. | 0.0186 |
| Metric | Raw | GPT-5.1 | Percent | |
|---|---|---|---|---|
| Brier score | 0.151657 | 0.142821 | ||
| Negation squared error | 0.046831 | 0.036621 | ||
| Multi-way partition squared error | 0.223655 | 0.180908 | ||
| Monotonicity violation | 0.018565 | 0.018164 |
Probabilistic Forecasting
Probabilistic forecasting predicts distributions or events with quantified uncertainty to support reliable decisions under uncertainty.
See all 245 papers