Overview

Proper scoring rules are losses for probabilistic predictions, such as the Brier score, log score, and continuous ranked probability score (CRPS); a strictly proper rule is minimized in expectation by reporting the true probabilities. Researchers use these rules to assess calibration and uncertainty, decompose forecast error, and develop guarantees for online forecasting and regret. They also tailor scores and training objectives to settings such as censored survival data, weather prediction, causal inference, and decisions with unequal costs. A key tension is that rules with the same truth-telling guarantee can produce different finite-sample model behavior and rankings, so the best score depends on both the data and the intended use of the forecast.

Latest papers

CardsList

34 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. Learning a calibrated decision head from logged bandit feedback

    Nov 11, 2026Ye YuanLanguage Model CalibrationConfidence Estimation in Language Models

  2. Estimating Spectrum-Free Information Bounds for Generative Surrogates Beyond Conditional Collapse

    Nov 5, 2026Chaitanya Dumbre, Sourabh DubeNeural Surrogate ModelingConditional Generative Modeling

  3. Strictly proper scoring rules for node selection tasks on graphs

    Nov 2, 2026Lorenz Hilfiker, Nathan Cincera Brack, Martin SterchiProper Scoring Rules

  4. Minimal Superior Penalties: Characterising and Calibrating Decision-Aware Proper Scoring Rules

    Nov 1, 2026Rouhollah AhmadianProper Scoring Rules

  5. Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation

    Oct 24, 2026Jinhao Zhou, Zhaohan Zhang, Xijian Xu +1Multi-Agent LLM SystemsLLM Agent Reliability

  6. A unified approach for estimation of unnormalized models via Stein operators

    Oct 21, 2026Mitsuki Nagai, Keisuke YanoParameter EstimationProper Scoring Rules

  7. AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

    Oct 15, 2026Liqin Ye, Rongzhi Zhang, Xin Liu +5Probabilistic ForecastingRL for LLM Agents

  8. Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting

    Sep 29, 2026Joonhyeong Park, Giung Nam, Hyungi Lee +3Probabilistic ForecastingProper Scoring Rules

  9. You Should Be Properly Scoring Your Odometry

    Sep 22, 2026Ola Rønning, Usama Saqib, Andrzej WąsowskiProbability CalibrationProper Scoring Rules

  10. How Proper Scoring Rules Shape LLM Forecasting

    Aug 28, 2026Benjamin Turtel, Paul Wilczewski, Kris Skotheim +2Probabilistic ForecastingProper Scoring Rules

  11. When do prophets profit in prediction markets?

    Jul 7, 2026Anri Gu, Nicole Kagan, Alec Sun +2Probabilistic ForecastingProper Scoring Rules

  12. Decision-Aware Training for Sample-Based Generative Models

    Jul 1, 2026Kornelius Raeth, Nicole LudwigCost-Sensitive LearningProbabilistic Forecasting

  13. Decision-Aligned Evaluation of Uncertainty Quantification

    Jun 25, 2026Annika Schneider, Tommy Rochussen, Joshua Stiller +1Uncertainty QuantificationProper Scoring Rules

  14. Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score

    Jun 23, 2026Giorgio Corani, Stefano Damato, Dario Azzimonti +1Model SelectionTime Series Forecasting

  15. Toward Simultaneously Optimal Regret in U-Calibration

    Jun 16, 2026Rafael Frongillo, Haipeng Luo, Nishant A. Mehta +1Follow-the-Perturbed-LeaderProbabilistic Forecasting

  16. Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

    Jun 8, 2026Jan Niklas Lettner, Hadeer El Ashhab, Benjamin SchäferProbability CalibrationElectricity Price Forecasting

  17. Proper Scoring Rules for Right-Censored Survival Data

    Jun 4, 2026Jef Jonkers, Glenn Van Wallendael, Luc Duchateau +1Survival AnalysisProbabilistic Forecasting

  18. Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference

    Jun 2, 2026Roman Plaud, Alexandre Perez-Lebel, Antoine Saillenfest +4Inverse Probability WeightingCausal Effect Estimation

  19. Proper Calibeating

    May 26, 2026Dean P. Foster, Sergiu HartProper Scoring RulesUncertainty Calibration

  20. Proper Scoring Rules for Agentic Uncertainty Quantification

    May 23, 2026Suresh Raghu, Satwik Pandey, Shashwat PandeyUncertainty QuantificationAgent Evaluation

  21. ECUASn_n: A family of metrics for principled evaluation of uncertainty-augmented systems

    May 19, 2026Lautaro Estienne, Erik Ernst, Matías Vera +2Uncertainty QuantificationProper Scoring Rules

  22. Calibeating for general proper losses: A Bregman divergence approach

    May 17, 2026Maximilian Fichtl, Cristóbal Guzmán, Nishant A. MehtaFunctional Bregman DivergencesProper Scoring Rules

  23. The Endogeneity of Miscalibration: Impossibility and Escape in Scored Reporting

    May 8, 2026Lauri Lovén, Sasu TarkomaMechanism DesignProper Scoring Rules

  24. Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search

    May 3, 2026Gerardo A. Flores, Yash Deshpande, Jannis R. Brea +1Proper Scoring Rules

  25. Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

    May 1, 2026Maksym Nechepurenko, Pavel ShuvalovAI Agent EvaluationForecasting Benchmarks

  26. ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

    Mar 31, 2026Jonas Landsgesell, Pascal Knoll, Tizian WenzelTabular RegressionTabular Foundation Models