Overview

Proper scoring rules are losses for probabilistic predictions, such as the Brier score, log score, and continuous ranked probability score (CRPS); a strictly proper rule is minimized in expectation by reporting the true probabilities. Researchers use these rules to assess calibration and uncertainty, decompose forecast error, and develop guarantees for online forecasting and regret. They also tailor scores and training objectives to settings such as censored survival data, weather prediction, causal inference, and decisions with unequal costs. A key tension is that rules with the same truth-telling guarantee can produce different finite-sample model behavior and rankings, so the best score depends on both the data and the intended use of the forecast.

Latest papers

34 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Mar 16, 2026

Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level A\mathcal A, the expected loss of an A\mathcal A-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels A⊆B\mathcal A\subseteq\mathcal B, a chain decomposition quantifies the information gain from A\mathcal A to B\mathcal B. Applied to classification with features X\boldsymbol{X} and score S=s(X)S=s(\boldsymbol{X}), this yields a three-term identity: miscalibration, a {\em grouping} term measuring information loss from X\boldsymbol{X} to SS, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.
Mar 9, 2026

Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

We study how the objective used to train or fine-tune a tabular foundation model affects its distributional predictions. Models such as TabPFN and TabICL provide forecasts of the full conditional distribution, but regression benchmarks including TabArena and TALENT assess them mainly through RMSE and R2R^2. Such assessments reward accurate conditional means while leaving the remaining distributional information unassessed. We therefore examine proper scoring rules as objectives for training and fine-tuning and as criteria for evaluating and ranking these models. Every strictly proper rule favors the true distribution at the population level, however, this shared optimum does not determine how learning proceeds with a finite sample. Our examples illustrate score-dependent preferences between forecasts, and our fine-tuning experiments show differences in predictive performance under different objectives. Experiments that fine-tune TabPFN and TabICL on several datasets reveal that changing the training objective can change performance across evaluation scores, with non-trivial interactions between the two. By connecting established methods from distributional regression, scoring-rule theory, and foundation-model adaptation, we show that selecting the score is part of specifying the model and can measurably alter the distribution it learns.
Mar 4, 2026

An intuitive rearranging of the Yates covariance decomposition for probabilistic verification of forecasts with the Brier score

Proper scoring rules are essential for evaluating probabilistic forecasts. We propose a simple algebraic rearrangement of the Yates covariance decomposition of the Brier score into three independently non-negative terms: a variance mismatch term, a correlation deficit term, and a calibration-in-the-large term. This rearrangement makes the optimality conditions for perfect forecasting transparent: the optimal forecast must simultaneously match the variance of outcomes, achieve perfect positive correlation with outcomes, and match the mean of outcomes. Any deviation from these conditions results in a positive contribution to the Brier score.
Feb 26, 2026

A Proper Scoring Rule for Virtual Staining

Generative virtual staining (VS) models for high-throughput screening (HTS) can provide an estimated posterior distribution of possible biological feature values for each input and cell. However, when evaluating a VS model, the true posterior is unavailable. Existing evaluation protocols only check the accuracy of the marginal distribution over the dataset rather than the predicted posteriors. We introduce information gain (IG) as a cell-wise evaluation framework that enables direct assessment of predicted posteriors. IG is a strictly proper scoring rule and comes with a sound theoretical motivation allowing for interpretability, and for comparing results across models and features. We evaluate diffusion- and GAN-based models on an extensive HTS dataset using IG and other metrics and show that IG can reveal substantial performance differences other metrics cannot.