Probabilistic Forecasting

Follow this topic Ask about this topic

Probabilistic forecasting estimates ranges, quantiles, event probabilities, or full predictive distributions rather than only a single future value. Methods include Bayesian and parametric models, ensemble post-processing and conformal calibration, and neural generative approaches such as diffusion and flow matching; language-model agents also forecast real-world events using evidence and probability aggregation. Researchers evaluate forecasts with proper scores such as CRPS and Brier score, alongside calibration, sharpness, and benchmarks in weather, energy, health, finance, and human mobility. A central challenge is balancing accuracy, reliable uncertainty, computational cost, and usefulness for downstream decisions, since strong average scores may hide failures in extremes or dependencies between outcomes.

Latest papers

230 papers37 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.

Aug 6, 2026

Signal or Spurious Cue? A Randomized Audit of Survey-Country Metadata in LLM Social Inference

Survey-country metadata can improve an LLM's forecast of an individual response when informative, yet the same cue may redirect the forecast when assigned at random. A within-record audit tests whether disclosing a random label's uniform, record-independent origin reduces its country-directed uptake, and whether verified survey country lowers held-out Brier loss. Independent population anchors and recorded human answers measure direction and consequence across five fixed API models, six countries, and seven development-selected targets. In the primary post-review 72-record panel, opaque and disclosed-random labels each produced country-direction shifts of 0.214. Paired attenuation was 0.0003 (95% CI [-0.0157, 0.0166]). Verified country reduced Brier loss by 0.040 (95% CI [0.024, 0.056]), while random-label regret included zero. A non-overlapping mixed-coverage consistency panel retained positive disclosed-random movement and verified utility, while attenuation remained uncertain. On the selected targets, verified metadata was useful in both panels, but disclosure did not reliably attenuate random-label uptake. PROV-FORECAST contains 14,400 paired item-level probability distributions from the corrected panel.

Yifan Lyu, Xinran Li, Jiaqi Qiao +1

Dalian University of Technology, China

Probabilistic Forecasting Causal Interventions in Language Models Language Model Auditing

Aug 5, 2026

Hybrid Probabilistic Zonotopes for Identifiable and Refinable Predictive Uncertainty

Probabilistic prediction heads in neural networks typically output either a Gaussian mixture or a single conformal region. Neither separates the distinct sources of uncertainty often present in real prediction tasks: a discrete choice among modes, bounded systematic drift within the chosen mode, and irreducible stochastic noise. We introduce the Hybrid Probabilistic Zonotope (HProbZ), an output head that represents these three sources as binary, bounded, and stochastic generators of a zonotope, and admits a closed-form likelihood by convolution. Sharing the bounded generator across prediction steps couples future predictions algebraically, so observing one step refines the predictive distribution at every remaining step in a single forward pass. We establish that the three generators are identifiable from the likelihood up to permutation, and that an HProbZ density is representationally distinct from any finite Gaussian mixture. The same shared structure provides analytic per-mode risk and distribution-free multi-modal conformal sets at inference time. Empirical analysis on representative prediction benchmarks supports the effectiveness of the design relative to same-encoder mixture baselines, while offering structural properties that mixture or convex-conformal predictors do not jointly provide.

Zhen Zhang, Amr Alanwar

School of Computation, Information and Technology, Technical University of Munich, Germany

Uncertainty Quantification Probabilistic Forecasting Conformal Prediction Uncertainty Decomposition

Aug 5, 2026

From Score Matrices to Football-Aware Match-State Simulation: An Auditable LLM Harness for Exact-Score Reranking

Football score forecasting combines a strong statistical core with a difficult contextual edge. Dynamic Poisson-family models estimate team strength, expected goals, and coherent score probabilities, but do not directly understand roles, tactical matchups, motivation, or how a first goal changes behaviour. Large language models (LLMs) can reason about such concepts, yet are not calibrated probability engines. We combine both components through an auditable information harness. This paper documents four iterations: V1, a dynamic score-driven Dixon-Coles baseline; V2, which maps LLM contextual ratings back into expected-goal parameters; V3, which replaces scalar correction with goal-by-goal simulations over a frozen score-candidate set; and V4, which adds shared first-breakthrough and post-goal cascade judgments, time-aware stopping, and deterministic tail candidates. The harness defines input semantics, supplies pre-match evidence, and constrains the LLM to an inspectable reasoning route. On a chronological replay of the first 150 matches of the 2025-26 English Premier League, V1 achieved 10.0% Top-1 and 26.7% Top-3 exact-score accuracy. V3 reached 12.0% and 30.0%, while V4 reached 14.7% and 30.7%. V4 increased candidate coverage from 77.3% to 84.7%, although no added tail candidate became a Top-3 exact hit. V1's native 1X2 distribution achieved 53.3% argmax accuracy, 0.9878 log loss, 0.5870 Brier score, and 0.2095 ranked probability score. These results are exploratory: the development slice is not an untouched benchmark, and temporal input isolation cannot exclude outcome memory in a closed LLM. The contribution is an auditable hybrid architecture, a clear design evolution, and negative findings showing where football-aware simulation does and does not improve score selection.

Shaopeng Liang

Independent Researcher

Probabilistic Forecasting LLM Reranking Sports Outcome Prediction Soccer Analytics

Aug 5, 2026

Real-time probabilistic tsunami forecasting via generative AI

Explicit onshore tsunami inundation forecasting can improve public risk awareness, but deterministically predicted inundation boundaries under highly uncertain conditions, such as near-field tsunamis generated by megathrust earthquakes, may falsely imply safety outside the boundaries. Consequently, current warnings primarily target coastal tsunami height, not onshore inundation. Although machine learning enables instant inundation predictions, they remain deterministic, lacking uncertainty quantification. Here, we develop a probabilistic ensemble model based on a conditional diffusion model (a type of generative AI) that reconciles accuracy with calibration. Validated with the 2011 Tohoku-oki earthquake data, our model faithfully tracks the postearthquake uncertainty decreasing over time while accurately predicting inundation depth and extent. Our framework shows that generative AI can shift tsunami forecasting from determinism to probabilism, providing a foundation for next-generation early warning.

Yusuke Oishi, Takashi Furumura, Fumihiko Imamura

Fujitsu Research, Fujitsu Limited, Kawasaki, Japan. · Earthquake Research Institute, The University of Tokyo, Tokyo, Japan. · International Research Institute of Disaster Science, Tohoku University, Sendai, Japan.

Uncertainty Quantification Probabilistic Forecasting Flood Inundation Mapping

Aug 4, 2026

SocietyBench: Forecasting Counterfactual Social-World Evolution

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

Zhenran Wang, Zhonghan Bian, Jinsong Li +1

Counterfactual Evaluation Forecasting Benchmarks Probabilistic Forecasting Language Model Calibration

Aug 4, 2026

WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.

Zhenran Wang, Zhonghan Bian, Jinsong Li +1

LLM Evaluation Forecasting Benchmarks Probabilistic Forecasting Sports Outcome Prediction

Aug 4, 2026

AI World Cup 2026: Benchmarking Large Language Models for End-to-End Football Tournament Prediction

Large language models (LLMs) are now regularly asked to forecast real-world events, but comparisons are often difficult because models receive different information, use different tools, and are evaluated under different rules. This paper reports the completed AI World Cup benchmark, in which ten LLM-based assistants made a single pre-tournament forecast of the entire 2026 FIFA World Cup. Every submission used the same tournament snapshot, prompt, JSON schema, and scoring procedure. The forecasts covered group-stage scores, group rankings, the knockout bracket, final placings, confidence values, and short explanations. After all 104 matches had been played, GPT-5.5 Thinking finished first with 744 points, followed by GPT-5.5 with 717, Gemini with 699, and Qwen 3.7 with 687. GPT-5.5 Thinking was also the only model to select Spain, which defeated Argentina 1--0 in the final, as champion. The final ranking was driven mainly by knockout performance: total score was strongly correlated with knockout points (

r=0.986

), but showed little relationship with group-stage match points (

r=0.055

), group-standing points (

r=-0.103

), or their combined pre-knockout score (

r=-0.054

). Match-level accuracy produced a different ordering. Claude Sonnet 4.6 correctly predicted the largest number of group-stage outcomes (63.89%) but placed sixth overall. Average self-reported confidence was also unrelated to either outcome accuracy (

r=-0.060

) or total score (

r=-0.067

). The results suggest that forecasting a complete tournament tests something different from predicting matches one at a time, while also showing how strongly a bracket-based leaderboard can depend on scoring design. The benchmark materials, raw responses, and scoring code are released to support replication and future extensions.

Jonaid Shianifar, Iias Faiud

AI World Cup Project

LLM Evaluation Forecasting Benchmarks Probabilistic Forecasting Sports Outcome Prediction

Aug 3, 2026

Probabilistic Deep Learning for Drought Forecasting: Role of Internal Climate Variability

Predicting drought risk is essential for anticipating impacts on water resources, agriculture, ecosystems, and climate adaptation planning. Yet drought forecasts remain uncertain because variability can substantially alter regional precipitation and evaporative demand. Treating this variability as unstructured noise ignores the fact that internal variability has spatial, seasonal, and temporal structure and thus contains information that can be used to improve drought forecasting. We propose a deep-learning-based forecasting framework for European drought prediction and extend it with an uncertainty-aware drought bound that explicitly incorporates internal forecast variability from a large climate model ensemble. This bound represents a physically plausible lower-tail trajectory of future drought conditions and marks how severe drought could plausibly become under an unfavourable realisation of internal variability, giving adaptation planning a conservative, risk-averse reference. We compare the proposed bound with a lower bound derived from reanalysis data only and show that our proposed ensemble-informed bound is better calibrated across most regions and seasons. This is specifically true during anomalously dry conditions, when historical reanalysis alone underestimates lower-tail drought risk. Our results show that internal variability should be treated as a forecast quantity in its own right. More broadly, large ensembles provide a practical way to transfer physically plausible climate variability into machine-learning drought forecasts, yielding risk-aware bounds that are more informative for drought assessment under shifting climate conditions.

Henri Funk, Cornelia Gruber, Göran Kauermann +2

Department of Statistics, LMU Munich, Munich, Germany · Department of Geography, LMU Munich, Munich, Germany

Climate Science Time Series Forecasting Probabilistic Forecasting Regression Prediction Intervals

Jul 28, 2026

Projected climate memory and inherited warm-tail risk in accelerated European summer warming

European summer warming reflects interactions among background change, persistent ocean--land--circulation states, and same-season variability. We develop an empirical reduced-dynamics framework that decomposes regional summer indicators into inherited slow-state memory, its predictable component, and contemporaneous innovation. Projection-operator theory motivates the decomposition, implemented with finite causal filters, ridge-regularised prediction, and logistic risk models. Using ERA5-derived summer indicators for 28 IPCC AR6 European sub-regions over 1950--2024, with validation on 2006--2024, we find that Mediterranean-state memory improves mean summer-temperature prediction relative to trend-only and ARX baselines. The gain over ARX is modest, while moving-average, exponentially weighted, and tempered filters contain similar annual information, indicating that the data identify useful slow-state memory more robustly than a unique kernel shape. Predictable-state reconstruction is ridge-sensitive and therefore treated diagnostically rather than as a forecasting model. The strongest result concerns warm-tail risk. In parsimonious logistic models, high accumulated Mediterranean memory raises predicted upper-tail event probability by about 8--11 percentage points for annual maximum summer temperature, warm-day frequency, and warm-spell duration at 1-, 3-, and 5-year horizons. Regional bootstrap intervals remain positive for all targets and horizons. Circular-shift placebos yield one-sided probabilities of approximately 0.05--0.14 and do not survive strict family-wise correction across nine tests, so the evidence is moderate rather than decisive. Overall, annual projected memory is not a universal short-horizon predictor, but a physically interpretable inherited risk-loading variable identifying years and regions predisposed to warm-tail outcomes.

Mauricio Herrera-Marín, Alex Godoy-Faúndez, Diego Rivera

Research Center on Sustainability and Strategic Resource Management (CISGER), Faculty of Engineering, Universidad del Desarrollo, Avda. Plaza 700, Santiago, Chile.

Probabilistic Forecasting Climate Modeling Weather Forecasting

Jul 27, 2026

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult. In particular, existing benchmarks are typically static and retrospective, and therefore cannot test how information is synthesized by LLMs to predict future events under uncertainty. We introduce LLM-SoccerArena (https://llm-soccerarena.com), a prospective live benchmark that evaluates how well LLMs forecast real-world sports events before the outcomes are known. LLM-SoccerArena provides (1) a prospective live benchmark protocol, (2) a public open-source platform, and (3) a factorial benchmark design together with tournament-related questions (e.g., which team will win). LLM-SoccerArena automatically records timestamped, schema-validated forecasts of unresolved events, together with prompts, model versions, tool traces, and costs. The factorial design varies along four dimensions: (1) model version (e.g., GPT-5.5, Claude Opus 4.8); (2) information access; (3) prompting strategy, and (4) forecast horizon. We demonstrate LLM-SoccerArena through a large-scale evaluation of the 2026 FIFA World Cup, in which seven LLMs generated forecasts for all 104 matches and 15 tournament-related questions. We provide a detailed analysis of model performance across information access, prompting strategy, and forecast horizon. As a result, LLM-SoccerArena provides new evidence about the forecasting performance of state-of-the-art LLMs. For example, LLMs with web access outperform those without, but only by a small margin (i.e., a 0.023 improvement in Brier score). Overall, LLM-SoccerArena provides a flexible, open-source platform for prospective benchmarking of unresolved events. LLM-SoccerArena will be continuously updated, and can be directly applied to future national and international tournaments and league competitions.

Jonas Schröder, Jonas Schweisthal, Oliver Müller +2

MCML & LMU Munich Munich, Germany · Paderborn University Paderborn, Germany · University of Cologne Cologne, Germany

LLM Evaluation Forecasting Benchmarks Probabilistic Forecasting Sports Outcome Prediction

Jul 23, 2026

Demographically-Informed Heat-Mortality Risk Curves via Risk Graph Neural Networks

Estimating heat-related mortality risk is a core task in environmental epidemiology, typically addressed with Distributed Lag Non-linear Models (DLNMs); interpretable exposure-response surfaces fitted to temperature-mortality time series. DLNMs are effective but ignore demographic and geographic context, despite well-established relevance to heat vulnerability. We propose Risk Graph Neural Networks (RGNNs), a hierarchical GNN encoder that uses granular census features to optimise DLNM coefficient vectors, preserving interpretable risk curve outputs while substantially improving predictive calibration. Evaluated across 10 regions of England and Wales on two unprecedented heat years, RGNN variants maintain both lower point-errors and near-nominal uncertainty coverage during the 2022 heatwave where baselines collapse.

Alex O. Davies, Eunice Lo, Rui Zhu

School of Geographical Sciences, University of Bristol, UK

Graph Neural Networks Probabilistic Forecasting Mortality Prediction

Jul 22, 2026

SPECTRA: State-Space Exogenous Context and Temporal-Frequency Resolution Architecture for Probabilistic Energy Forecasting

Modern power systems increasingly require probabilistic forecasts amid interacting uncertainties from renewable intermittency, flexible demand, market volatility, and weather-dependent generation. However, existing methods often treat multi-scale decomposition, exogenous-variable alignment, and probabilistic output as separate steps, obscuring how predictable structures and uncertainty-bearing fluctuations jointly shape the forecast distribution. This paper proposes a state-space exogenous-context and temporal-frequency resolution architecture for general probabilistic energy forecasting. Its central premise is that trend-periodic components primarily determine the baseline trajectory, whereas high-frequency residuals and external perturbations govern the spread and asymmetry of forecast uncertainty. Accordingly, the architecture adaptively separates deterministic and residual streams, aligns exogenous context with both, refines the deterministic backbone through multi-resolution spectral-temporal state-space modeling, and estimates ordered quantile boundaries from their complementary representations. Experiments on load, price, solar, and wind forecasting achieve the best continuous ranked probability score in 14 of 18 settings, reducing average CRPS by 5.74% and upper-tail quantile risk by 7.27% over the strongest baselines. These results support deterministic-stochastic separation as an effective design principle for general probabilistic energy forecasting.

Hang Ye, Xinyan Jiang, Yuedong Shi +5

Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai 201210, China · University of Chinese Academy of Sciences, Beijing 101408, China · School of Physical Science and Technology, Shanghaitech University, Shanghai 201210, China +2

Probabilistic Forecasting Electricity Load Forecasting Multiscale Time-Series Modeling Probabilistic Time Series Forecasting State Space Models Power Systems

Jul 21, 2026

Probabilistic Physics-Aware Machine Learning Predictions of Electric Truck Energy Consumption with Field Data

In this work, we incorporate first principle physics into the construction of data-driven methods by considering a model that accounts for the different sources of energy losses during vehicle operations. Our results show that Bayesian linear regression based on this physics-aware model can improve the reliability of the expected energy consumption, as compared with standard linear regression. Further, it is shown that more complex machine learning models such as neural networks and gradient boosted regression trees, based on the same physical model, can further improve the accuracy in energy forecasting and significantly outperform standard versions of the same machine learning models. In addition to point predictions of the energy consumption, we develop a framework for estimating the corresponding uncertainty in the form of predicted standard deviation. Our results show that all of the models learn to estimate the uncertainty reasonably well.

Hannes Nilsson, Rafael Basso, Balázs Kulcsár +1

Volvo Technology AB Gothenburg, Sweden · Department of Electric Engineering Chalmers University of Technology Gothenburg, Sweden · Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden

Uncertainty Quantification Probabilistic Forecasting Physics-Informed ML

Jul 20, 2026

Censoring-Aware In-Context Learning for Generalized Supplier Lead Time Estimation in Supply Chain Planning

Supplier lead time forecasting is a central input to material requirements planning, inventory optimization, and supply chain risk management. However, many industrial lead time datasets are naturally right-censored: at the time forecasts are required, some orders have not yet arrived. Standard regression and classification approaches discard this information, while conventional survival models require task-specific modeling. We propose LeadTime-ICL (LT-ICL), a censoring-aware in-context learning model for probabilistic lead time forecasting. LT-ICL combines a transformer backbone with a conditional normalizing-flow head, producing a full predictive distribution over lead times. The model is pretrained on synthetic right-censored lead time tasks, enabling in-context adaptation to new industrial datasets without task-specific parameter updates. We provide theoretical support for this formulation by showing that excess CRPS is bounded by prior misspecification and amortized approximation errors, providing clear direction for improving forecasting performance. We evaluate LT-ICL on 24 proprietary supply-chain datasets spanning seven industries. LT-ICL achieves the lowest point-forecasting error on 15 of the 24 datasets, and the lowest probabilistic forecasting error on 14 datasets, yielding the best average rank across both. These results support right-censored probabilistic forecasting as a practical formulation for supplier lead time prediction and demonstrate that pretrained in-context models can provide accurate, low-adaptation-cost forecasting for industrial planning systems.

Christopher Wang, Sebastien Ouellet, Behrouz Haji Soleimani +1

Department of Electrical and Computer Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada · Kinaxis Inc., Ottawa, ON K2K 3J1, Canada

Survival Analysis Time Series Forecasting Probabilistic Forecasting In-Context Learning Supply Chain Management

Jul 20, 2026

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across systems, similar result accuracy can mask larger differences in detailed predictions. Four systems predicted champion Spain, and two of them also recovered the exact final pairing. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, predictions and evaluation scripts will be publicly released.

Zhaokai Wang, Tianlin Gui, Jiayuan Rao +3

Shanghai Jiao Tong University · Shanghai, China · Nanjing University +4

LLM Evaluation Forecasting Benchmarks Probabilistic Forecasting Sports Outcome Prediction Agent Benchmarks

Jul 20, 2026

FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches

We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events. For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models -- Claude Opus 4.8, ChatGPT (GPT-5.5, high reasoning), Gemini 3.1 Pro, and Grok (Expert Mode) -- ran an identical search-act-reflect loop: gather evidence with a web tool, commit to a 1X2 (team-A win / draw / team-B win) distribution and a virtual 100-USD bet, and, after the match, reflect given only the final score. Because every match kicked off after the models' training cutoffs, the benchmark is contamination-free by construction. Crucially, we pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market -- collected as per-match 1X2 odds, giving an economically grounded baseline and letting us score not just what an agent predicts but what it does with money. The release contains 416 forecasts and 414 reflections with verbatim reasoning, ground truth (including penalty shootouts), odds, and a reproducible evaluation suite. A reference evaluation surfaces findings that raw accuracy hides: the four agents issue an identical top pick in 92% of matches and none beats the market's Brier score; indeed, a naive flat stake on the market favorite out-earns all four agents. Yet the agents diverge sharply as decision-makers: betting return-on-investment ranges from -18% to +10%, fading the market is unprofitable for all four, the share of forecasts that cite the market ranges from 12% to 100%, and self-reported error rates on wrong picks range from 36% to 86%. The benchmark thus measures calibration, decision quality, and self-knowledge -- axes on which frontier models differ even when their predictions do not. Data and code: https://github.com/graphuofm/FIFA2026LLM

Jiacheng Ding, Cong Guo, Jason Xu

University of Memphis Memphis, Tennessee, USA · QuantaInsight Seattle, Washington, USA

LLM Agent Evaluation Forecasting Benchmarks Probabilistic Forecasting Language Model Self-Assessment Sports Outcome Prediction

Jul 14, 2026

Learning-based Probabilistic Load Forecasting with Post-hoc and In-model Uncertainty

Smart-building load forecasters are often trained offline on dense, multivariate, high-frequency data, but deployment may provide only hourly, feature-limited inputs. Missing features must then be reconstructed, and their errors can propagate through the model. If this input uncertainty is not reflected, prediction intervals may become miscalibrated, affecting demand-response scheduling. Our work examines where uncertainty should be placed once inference inputs are reconstructed. We develop a unified one-day-ahead probabilistic forecasting framework that aligns temporal resolution, reconstructs the unavailable inputs, and derives causal features, and we compare a modular post-hoc residual-quantile scheme with an integrated in-model quantile-learning scheme. The comparison uses three mid-scale Deep Learning (DL) backbones: recurrent, hybrid recurrent, and attention-based Temporal Fusion Transformer (TFT) models, under identical inputs, forecasting horizon, preprocessing rules, and training budgets. Results show that uncertainty placement is backbone-dependent. Integrated quantile learning is most reliable with the TFT, yielding 2.2-3.6% MAPE and 28-83W RMSE on the labeled test window, while producing intervals about 5x narrower than the modular intervals at the closest-to-nominal coverage level. Diebold-Mariano tests support the TFT ranking and the mixed behavior of the recurrent backbones. A reconstruction-sensitivity test shows that reconstructed inputs increase the Quantile Score (QS) by 106% while interval width remains nearly unchanged, indicating that the model does not automatically absorb reconstruction-induced uncertainty. Robustness checks against non-DL baselines and seasonal hold-out weeks support this ranking. Our results expose the limits of post-hoc residual quantiles when inference depends on reconstructed inputs.

Sarah Al-Shareeda, Gulcihan Ozdemir, Heung Seok Jeon

aComputer System and Technology Department, Faculty of Computer Science and Information Technology, University of Malaya, Malaysia · bInformatics Institute, Istanbul Technical University, Turkey · cComputer Engineering Department Konkuk University South Korea

Quantile Regression Time Series Forecasting Probabilistic Forecasting Electricity Load Forecasting Uncertainty Calibration Probabilistic Time Series Forecasting

Jul 13, 2026

From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth

Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.

Aleh Manchuliantsau

Independent Researcher

LLM Evaluation Probabilistic Forecasting Automated Evaluation

Jul 10, 2026

Manifold Constrained Conformal Prediction for Spatial Events

We introduce a new conformal prediction method that constructs calibrated prediction sets over collections of spatial events, such as tropical cyclone genesis and earthquake locations. Forecasting natural hazards has become increasingly important, due to their significant economic impact, and quantifying the uncertainty of predictions is critical for accurate risk assessment. Our approach works by representing spatial point clouds as empirical measures so that we can score them using (sliced) Wasserstein distance, then constraining the resulting distribution-valued prediction set to be supported only near the training data manifold. We derive a coverage lower bound for the intersected sets and show that, in practice, this gap can be made small through a simple data-adaptive selection criterion. Because the resulting set is not analytically tractable, we introduce a modified flow-based sampling procedure, which allows us to represent and apply these prediction sets in practice as ensembles. Numerical experiments on synthetic data, tropical cyclone genesis, and earthquake occurrences show that our method achieves near-nominal coverage, with significantly lower energy distance and manifold distance than highest predictive density region (HDR) baselines along with generative model baselines.

Collin Nill, Trevor Harris, Jason Adams

Department of Statistics University of Connecticut Storrs, CT 06269 · Sandia National Laboratories Albuquerque, NM 87123

Set Prediction Probabilistic Forecasting Conformal Prediction Spatiotemporal Forecasting

Jul 10, 2026

Global Merger-Arbitrage Forecasting with Language Models

We present a language-model forecasting system for merger arbitrage, a specialized high-stakes financial setting in which the task is to predict the outcome of announced M&A deals. Unlike prior work on judgmental forecasting with LLMs, which has focused on broad mixed-topic benchmarks and short context such as news snippets, we study a setting that requires long-context reasoning over hundreds of pages of technical documents. Our system combines expert-guided context engineering with finetuning on hindsight-guided reasoning traces derived from historical deals. Given an announced deal, it outputs a probability distribution over three mutually exclusive outcomes: closing at announced terms, a higher bid, or deal termination. On an out-of-sample set of more than 400 large deals spanning 42 countries, our finetuned system achieves the best performance of any method we evaluate, reducing class-balanced Brier score to 0.151. This is 24% below calibrated market-implied probabilities, 19% below XGBoost, and 25-42% below frontier language models. These results, together with ablation studies, show that LLM-based forecasting can succeed in specialized, long-context financial workflows, with hindsight-based supervision and expert-designed context playing a critical role.

Hinal Jajal, Michal Mucha, Charles Sweat +3

Balyasny Asset Management

Long-Context Language Model Inference LLM Fine-Tuning Financial Forecasting Probabilistic Forecasting Finance

Jul 7, 2026

When do prophets profit in prediction markets?

Prediction markets aggregate dispersed beliefs into prices that act as probabilistic forecasts of uncertain events. Classical theory establishes a clean equivalence between forecasting accuracy and trading profit, but only for the specific automated market maker (AMM) design. However, the largest exchanges today are based on central limit order books in which informed forecasters routinely lose money while uninformed strategies can profit on simple heuristics. We resolve this discrepancy by establishing a formal equivalence between predictive accuracy and profitability. For any strictly proper scoring rule

S

, we exhibit a "proper" betting strategy that depends only on the forecaster's prediction

\mathbf{p}

and the market price

\mathbf{q}

, and earns positive expected profit whenever

\mathbf{p}

outperforms

\mathbf{q}

under

S

and the market has sufficient liquidity. Moreover, this proper betting is essentially the only strategy with such robust profitability guarantee. The proof rests on a decomposition of expected profit that strictly generalizes the classical AMM guarantee and also explains how strategies can profit without an accuracy edge. Empirically, across thousands of forecasts by AI models, proper betting is the only strategy that reliably converts accuracy into profit, and we further identify systematic forecasting personas and show how the optimal proper strategy varies across them. A month-long live deployment on Kalshi achieves

+80.33\%

return on investment with a Sharpe ratio of

3.35

.

Anri Gu, Nicole Kagan, Alec Sun +2

University of Chicago · Kalshi Research · New York University, Shanghai

Probabilistic Forecasting Proper Scoring Rules Finance Algorithmic Trading Prediction Markets

Jul 6, 2026

AIFS-SUBS: Extending Data-Driven Forecasting to Sub-Seasonal Timescales

Data-driven models now rival numerical weather prediction in the medium range, but extending them to sub-seasonal lead times raises challenges absent at shorter horizons. Errors accumulate over long autoregressive rollouts, systematic biases grow with lead time, and several years of data must be held out for independent verification, even though machine-learning models otherwise benefit from longer training records. To address these challenges, we adapt ECMWF's AIFS-CRPS medium-range model. AIFS-SUBS adopts a 24h autoregressive time step to reduce error accumulation, adds stratospheric levels and top-of-atmosphere thermal radiation as predictors, and reserves 2007--2011 as an independent verification window. We evaluate two config-durations: AIFS-SUBS, fine-tuned on operational analyses, and AIFS-SUBS-ERA5, trained on ERA5 alone. Across weeks 2--6, AIFS-SUBS matches the operational Integrated Forecasting System (IFS) in probabilistic skill while reducing systematic biases. For the convective (OLR) component of the Madden--Julian Oscillation (MJO), AIFS-SUBS extends skilful forecasts (correlation > 0.5) by eight days relative to the IFS, while matching or exceeding the IFS for the full multivariate RMM index. AIFS-SUBS also reproduces the observed MJO modulation of tropical cyclone activity comparably. Stratospheric skill is particularly strong with AIFS-SUBS reproducing sudden stratospheric warming (SSW) frequency and surface impact. In the AI Weather Quest, AIFS-SUBS-ERA5 attains a variable-averaged ranked probability skill score slightly ahead of the IFS at weeks 3 and 4. At inference, AIFS-SUBS uses about 200 times less energy than the IFS, opening the door to much larger real-time ensembles. AIFS-SUBS is ECMWF's first machine-learning model targeted at sub-seasonal time-scales.

Jakob Schloer, Steffen Tietsche, Christopher D. Roberts +6

1European Centre for Medium-Range Weather Forecasts (ECMWF)

Probabilistic Forecasting Weather Forecasting Autoregressive Modeling

Jul 5, 2026

Exploring Convolutional Neural Processes for Weather Downscaling

Global reanalysis products such as ERA5-Land provide spatially complete weather fields but at resolutions too coarse for local applications, particularly in mountainous regions where temperature can vary by several degrees over short distances. This project investigates Convolutional Conditional Neural Processes (ConvCNPs) for statistical downscaling of daily maximum temperature from the ~11km resolution ERA5-Land grid to ~1km resolution over Switzerland, building upon the architecture of Vaughan et al. (2022) and adapting it to the topographically complex Swiss domain with high-resolution elevation features from the swisstopo DHM25. The best model, trained on ten years of data (2014-2023) with five-fold temporal cross-validation, achieves a mean absolute error of 1.31 Celsius and a CRPS-based skill score of 0.524 relative to bilinear interpolation, reducing the expected prediction error by more than half. An ablation study reveals that the elevation MLP is the indispensable component - without it, the model diverges entirely - while explicit seasonal features and Topographic Position Index provide secondary benefits. Under sparse on-grid input the model degrades gracefully, maintaining positive skill down to approximately 10% of the input grid; however, zero-shot deployment on off-grid station observations does not achieve positive skill at any density tested. All configurations exhibit severely overconfident uncertainty estimates, a structural limitation of the Gaussian likelihood training objective. These results demonstrate that ConvCNPs are a viable and effective approach to climate downscaling in complex terrain, and identify uncertainty calibration and native support for non-gridded input as the key challenges for operational deployment.

Francisco Passos

ETH Zürich

Probabilistic Forecasting Climate Downscaling Weather Forecasting

Jul 3, 2026

Joint distribution of upstream runoff governs downstream river-discharge prediction uncertainty in distributed ML models

Uncertainty quantification of hydrological predictions is necessary to inform operational decisions. Recent generative machine-learning methods have advanced probabilistic streamflow prediction, but have remained confined to lumped models that predict a basin outlet directly. At the same time, deterministic LSTM runoff models are increasingly applied at grid or catchment scale and routed through river networks to produce spatially continuous, physically consistent discharge fields. This technical note argues that moving probabilistic prediction from lumped to distributed models introduces a specific new requirement: the joint distribution of upstream runoff generation must be sampled jointly. In lumped inference, the model predicts the outlet distribution directly and can modulate spread from basin attributes. In distributed inference, downstream discharge is obtained by routing many upstream runoff predictions, so independent local sampling averages uncertainty away. Using Japan as a case study, we train two probabilistic basin-scale runoff LSTMs and route their runoff through a Hayami routing scheme. Randomly matching upstream ensemble members produces severely under-dispersed downstream ensembles, whereas a simple quantile matching strategy restores much of the spread of the direct basin-scale reference. The shift from lumped to distributed probabilistic hydrology therefore requires explicit attention to the spatial joint structure of runoff uncertainty.

Karan Ruparell, Tristan Hascoet, Takemasa Miyoshi +4

University of Reading, Department of Meteorology, Reading, UK · European Centre for Medium-Range Weather Forecasts (ECMWF), Reading, UK · RIKEN Center for Computational Sciences, Kobe, Japan +3

Streamflow Forecasting Uncertainty Quantification Time Series Forecasting Probabilistic Forecasting Hydrology

Jul 2, 2026

Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an objective, externally resolved benchmark, this pilot shows that the value of human-AI collaboration depends on a specific, measurable form of human capital. Analyzed at the level of the individual forecaster, hybrid performance is trimodal: most people either deferred to the model (matching it) or used it to rubber-stamp a prior guess (performing worse than the model alone), while a minority engaged in genuine complementary reasoning and reached accuracy matching or even exceeding (i.e., lower error than) the market itself. Collaborative traits (perspective-taking, intellectual humility, and curiosity) rather than raw cognitive ability or model benchmarks, distinguished who reached that mode. The results are preliminary but statistically robust, and motivate a pre-registered replication now in preparation.

Vivienne Ming

The Human Trust; Possibility Science; UCL Global Business School for Health

Human-AI Complementarity Probabilistic Forecasting Human-AI Collaboration

Jul 2, 2026

Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry

Multi-agent systems are increasingly used for forecasting future events, as deliberation among multiple LLMs is believed to improve reasoning and calibration. Yet existing approaches overlook a critical design choice: what information each agent receives. When all agents are given identical evidence, deliberation collapses into herding rather than genuine belief revision, leaving multi-agent systems little better than a single agent. We identify this as a fundamental gap and propose designed information asymmetry to close it: by partitioning evidence into shared public and disjoint private subsets, each agent holds exclusive knowledge that can only reach others through deliberation. We theoretically show that this decomposition reduces inter-agent error correlation, and instantiate it in InfoDelphi, a framework combining relevance-aware evidence routing, rationale-based iterative deliberation, and confidence-weighted aggregation. On PolyGym, a benchmark of 375 binary forecasting questions derived from real-world prediction markets, InfoDelphi outperforms the strongest single-agent and multi-agent baselines by 12--18% in Brier score and 4--8 percentage points in accuracy. More detailed experiments confirm that removing information asymmetry eliminates most deliberation gains, establishing diversity of input as the key enabler of effective multi-agent reasoning.

Yuante Li, Yicheng Tao, Kate Zhang +3

School of Computer Science, Carnegie Mellon University · College of Engineering, Carnegie Mellon University

Probabilistic Forecasting Multi-Agent Debate Multi-Agent Reasoning Multi-Agent Deliberation

Jul 1, 2026

Decision-Aware Training for Sample-Based Generative Models

Sample-based generative models are increasingly used for probabilistic forecasting in high-stakes decision settings, yet their training objectives are blind to the decision maker's cost structure. These models are commonly trained with strictly proper scoring rules, such as the energy score, which allocate their training signal in proportion to data density, with no awareness of where forecast errors are most costly for downstream decisions. We therefore propose decision-aware training for sample-based generative models, augmenting the energy score objective with a differentiable decision loss that directly penalises the cost incurred by acting on the model's forecast. This combined loss is theoretically grounded, as the decision loss is itself a proper scoring rule. We validate our method on one synthetic and two real-world tasks, showing targeted improvements in cost-sensitive regions while retaining full probabilistic forecasts.

Kornelius Raeth, Nicole Ludwig

T ubingen AI Center, University of T ubingen, Germany · University of Augsburg, Germany

Cost-Sensitive Learning Probabilistic Forecasting Decision-Focused Learning Proper Scoring Rules Generative Modeling

Jul 1, 2026

Towards Metric-Agnostic Trajectory Forecasting

Accurate trajectory forecasting of surrounding traffic participants is a core capability for autonomous driving, enabling vehicles to anticipate behavior and plan safe maneuvers. We observe that current state-of-the-art forecasting models on Argoverse 2 and the Waymo Open Motion Dataset tailor their training objectives to the different benchmark metrics. Because these metrics encourage conflicting behavior, we propose a paradigm change for trajectory forecasting: training models with metric-agnostic probabilistic objectives and treating metric optimization as a downstream task applied to the predictive distribution. Concretely, we introduce Trajectory Distribution Evaluation (TraDiE) policies, metric-specific policies that map a predictive distribution to the set of

K

trajectories and confidences required by trajectory forecasting metrics. We evaluate this framework by introducing DONUT-NLL, which adapts the training objective of the state-of-the-art trajectory forecasting model DONUT to directly optimize the predictive distribution. Using our policies, DONUT-NLL achieves state-of-the-art results on all metrics of the Waymo motion prediction benchmark.

Markus Knoche, Daan de Geus, Bastian Leibe

RWTH Aachen University, Germany · Eindhoven University of Technology, Netherlands

Autonomous Driving Benchmarks Multi-Agent Trajectory Prediction Probabilistic Forecasting Trajectory Prediction Probabilistic Trajectory Prediction

Jul 1, 2026

Neural Network-Based Estimation of Time-Dependent Parameters in AR(p) Processes

We investigate a forecasting framework based on a simple discrete-time dynamic model with coefficients varying in time. The parameters of the model are recovered within a deep learning framework, which makes it possible to retain a transparent parametric structure while simultaneously accounting for complex and nonstationary patterns in the observed phenomenon. Our analysis covers two specifications of the noise process. Besides the standard Gaussian setting, we also consider Laplace-distributed noise, which can offer a more adequate description in the presence of heavier tails and sharper local fluctuations. For both cases, we formulate the predictive scheme of the model and analyze the associated uncertainty quantification, including the construction of prediction intervals. The results illustrate that a relatively simple model, when combined with time-dependent parameter estimation, can serve as a mathematically tractable and practically flexible tool for forecasting complex dynamics under different noise assumptions. The general model is stated for TVAR(

p

), while the prediction-interval formulas and the numerical experiments are developed for the TVAR(1) case.

Agnieszka Kopeć, Paweł Przybyłowicz, Martyna Wiącek

Parameter Estimation Probabilistic Forecasting Non-Stationary Time Series Forecasting Regression Prediction Intervals Autoregressive Modeling

Jun 30, 2026

TRIE: An Evaluation Framework for Stochastic PDE Surrogates

Many scientific systems exhibit uncertainty from stochastic forcing, unresolved degrees of freedom, or imperfect observations, making reliable surrogate forecasting fundamentally distributional rather than pointwise. For such systems, deterministic neural surrogates fail to capture statistical measures and forecast uncertainty. We introduce TRIE, an evaluation framework for stochastic PDE surrogates that asks whether models reproduce invariant measures, provide trustworthy predictive uncertainty, and scale to efficient probabilistic generation. We demonstrate TRIE on two stationary chaotic spatially extended SPDEs, stochastic Kuramoto--Sivashinsky and stochastic Kolmogorov flow, across 11 parameter values. Our evaluation shows that standard pointwise-trained neural surrogates can produce plausible short rollouts while failing to match long-time statistical structure. Approximate uncertainty methods such as Monte Carlo dropout and heteroscedastic Gaussian likelihoods produce stochastic forecasts, but are often miscalibrated and overconfident under temporal and spatial uncertainty diagnostics. Across these criteria, generative models provide the most consistent performance, accurately capturing invariant measure statistics and achieving the lowest CRPS in all reported probabilistic settings. Finally, we show that latent generative models with automatic dimension discovery retain much of this statistical fidelity while reducing Kolmogorov inference time by roughly

12\times

. We release our code and data at https://github.com/scailab/TRIE-SPDE-Bench to support reproducible evaluation of stochastic PDE forecasting models.

Bharat Srikishan, Javier E. Santos, Nikhil Muralidhar +1

Stevens Institute of Technology · Los Alamos National Laboratory

PDE Surrogate Modeling Probabilistic Forecasting Generative Modeling Stochastic PDEs Uncertainty Calibration