Training a classifier means letting a validation metric decide which model is kept and when training stops. A natural requirement on that metric is superiority: every correctly classified forecast should score better than every incorrectly classified one, however close the underlying probabilities are. Neither the Brier score nor the logarithmic score satisfies this on its own. We show that superiority is governed by the separation between two endpoints -- the supremum of the score on correct predictions and the infimum on incorrect predictions -- and derive both in closed form for each score. Their difference is the infimal penalty, the greatest lower bound on the penalty needed to restore superiority. Prior work restores superiority with a penalty calibrated from the first endpoint alone; for the Brier base, our infimal penalty is smaller than that published penalty by a factor ranging from 4.00 at three classes to 2.11 at twenty. We further embed the published and infimal penalties in a one-parameter family and show that, under a single-valued tie convention, every member of the family is strictly proper. In a pre-specified two-dataset case study, mean selected-model macro-F1 rises with the penalty in the reported ten-class regimes, though not distinguishably from noise once seed/selection-mode nesting is accounted for; a fifth regime at forty-seven classes does not reproduce even this descriptive pattern. The intended no-superiority baseline is algebraically a rescaling of the family under test, so the experiment cannot attribute the observed pattern specifically to superiority. When superiority itself is the objective, any penalty strictly above the infimal value suffices; choosing a larger penalty is a separate calibration trade-off rather than a requirement for the guarantee.
Figures & tables
Figure 1: The range of sum-form Brier scores attainable on correctly classified forecasts, [0,Σψ] , and on incorrectly classified ones, [Σξ,→) , at c=13 . The two ranges overlap (shaded); the overlap has width λ∗=Σψ−Σξ=0.423 , the infimal penalty that separates them. Σψ=0.923 and Σξ=0.500 follow from the closed forms established in Section 4 ; the hatched bar is drawn open-ended since the incorrect range is unbounded above.
Convention
ω=1
ω=2
ω=3
A (permissive)
correct
correct
incorrect
B (single-valued, 1≺2≺3 )
correct
incorrect
incorrect
C (ties-to- ξ )
incorrect
incorrect
incorrect
Table 1: Whether p=(0.5,0.5,0) counts as a correct prediction of the true class ω , by convention.
Work
What it targets
Pointwise ordering guarantee
Propriety preserved
Closed-form penalty threshold
Kull and Flach (2015)
Decomposes a proper rule into calibration, refinement, adjustment, and grouping terms
Not addressed
N/A (decomposes an already-proper rule)
No
Resin (2023)
Elicitability bridge, single-class to full predictive distributions
Not addressed
Not the focus
No
Hu and Kash (2025)
Weighted sum of proper scoring rules for sub-properties
Not addressed
Weighting is designed to preserve it
No
Shahroudi et al. (2025)
Learned proxy aligning an upstream metric with downstream decision value (regression)
Reweighting framework for cost-sensitive binary classification
Not addressed
Not propriety-preserving
No
Table 2: Positioning against the closest prior approaches, on the dimensions this paper’s own contribution combines.
Figure 2: The probability simplex at c=3 . Dashed lines mark the three argmax tie loci {p:pi=pj} . The uniform vector attains supψSBS=0.667 (Proposition 2 ); a pairwise tie at the value 0.5 attains infξSBS=0.500 (Proposition 3 ). Both marked points sit exactly on the tie set, a measure-zero subset of the simplex — precisely where the tie conventions disagree with each other about attainment.
Figure 3: Expected sum-form Brier-penalised score at c=3 , Q=(0.5,0.3,0.2) , under Convention A, for a truthful forecaster ( P=Q ) and a forecaster who always reports uniform ( P=u ). Both curves are exact evaluations of the expected score on a λ grid, not a simulation. The uniform report incurs zero expected penalty at every λ under this convention, so its curve is flat; the truthful curve rises and crosses it at λA(Q)≈0.093 , found by linear interpolation between grid points, not by evaluating a separate closed-form formula. Shaded region: λ>λA(Q) , where misreporting scores strictly better than truthful reporting. The published penalty λ=0.667 (dotted) lies inside this region.
Brier ( PBS )
Log ( PLL )
Source def.
Eq. 42, λ=Σψ=cc−1
Eq. 43, λ=Σψ=logc
Published λ
0.923
2.565
Infimal λ∗
0.423
1.872
Ratio, published/ λ∗
2.18×
not tabulated
Implementation
Algorithm 2
Algorithm 3
Impl. vs. published
Mean-form; same λ
Matches published exactly
Table 3: The calibration facts of this section, gathered in one place, at c=13 (the value used throughout Figures 1, 2, and 4).
Figure 4: The admissible range for λ at c=13 : zero, the infimal penalty λ∗=0.423 , the published penalty 0.923 , and the released implementation’s sum-form-equivalent penalty 12.000 . The axis position of each value is λ , not λ itself, compressing the wide gap to the implemented penalty for display; axis labels give the true values. The superiority region λ>λ∗ is shaded, and the error-rate limit λ→∞ (equivalent to monitoring error rate directly) is marked off the right edge rather than placed at a finite position.
c
λ∗
Published λ
Ratio
Implemented λ
Ratio
3
0.167
0.667
4.00×
2.0
12.00×
10
0.400
0.900
2.25×
9.0
22.50×
13
0.423
0.923
2.18×
12.0
28.36×
20
0.450
0.950
2.11×
19.0
42.22×
47
0.479
0.979
2.04×
46.0
96.09×
Table 4: How far each Brier-base penalty sits from λ∗ , across the class counts already used elsewhere in this paper.
Grid point
Brier λ
Log λ
What it is
zero
0
0
No penalty; the base rule alone
critical ( λ∗ )
0.400
1.609
The infimal penalty (Corollary 4 , theory part 2)
supercritical
0.400001
—
λ∗+10−6 : strictly admissible
published
0.900
2.303
The source paper’s own calibration
implemented
9.000
n/a
The exact magnitude the source paper’s own experiments ran under, on the Brier base; the log base has no comparable value to report (relation to published rules)
limit
50.0
50.0
Approximates λ→∞ , i.e. monitoring error rate directly
Table 5: The λ grid at c=10 . Supercritical sits immediately above λ∗ , since Theorem 2 (Section 5 ) makes λ∗ itself convention-dependent for attainment, and testing the boundary needs a point unambiguously inside the admissible set under every convention.
Regime
What changes
Why
full
Nothing — full training data, full network capacity
The dataset’s natural difficulty
subset
Training data reduced
Fewer examples, more room for near-ties
noisy
Symmetric label noise injected
A different, independent way to create near-ties
hard
Reduced data and label noise together
Both pressures combined
Table 6: The four regimes, crossed with the λ grid for each dataset.
Baseline
What it measures
Purpose
Monitor macro-F1 directly
Select the checkpoint with the best macro-F1, full stop
Upper bound on macro-F1-driven selection
Monitor error rate directly
Select on error rate alone
The λ→∞ limit of the family
Weighted mixture Mw
Brier and error rate, blended at the same weights Sλ sweeps, but without superiority
Meant to isolate superiority from re-weighting; turned out not to (Section 8.6 )
Table 7: The three baselines.
Outcome
Macro-F1 past λ∗
Calibration and NLL
What it would mean
flat
Statistically flat
Still degrade
Superiority is the active mechanism; the published penalty spends calibration sensitivity for no macro-F1 gain
rising
Keeps rising, even past published
Still degrade
Superiority is incidental to whatever benefit the penalty gives; monitor error rate or macro-F1 directly instead
peaked
Peaks at an interior λ
Still degrade
A genuine trade-off; the contribution is a λ -selection rule, not one recommended value
Table 8: The three pre-specified outcomes.
Dataset/regime
Macro-F1
Informative
Near-tie rate
Penalty-fire rate
mnist/full
0.9953
no
0.0%
0.5%
mnist/subset
0.9818
no
0.0%
1.8%
mnist/noisy
0.9815
no
0.3%
1.8%
mnist/hard
0.8830
yes
1.2%
11.6%
cifar10/full
0.9144
no
0.1%
8.6%
cifar10/subset
0.6080
yes
2.8%
38.9%
Table 9: Every (dataset, regime) cell: mean validation macro-F1 (the informativeness criterion), whether it clears the 90% threshold, the near-tie rate (fraction of validation samples within 0.01 of a top-two tie), and the penalty-fire rate (fraction of validation samples on which the λ penalty actually changes which class is credited).
Base
Mean diff.
Median
SD
95% CI ( t )
95% CI (bootstrap)
Cells
Brier
−0.0036
0.0000
0.0067
[−0.0092,0.0020]
[−0.0090,0.0000]
1/2/5
Logarithmic
−0.0034
−0.0001
0.0054
[−0.0078,0.0011]
[−0.0071,−0.0005]
1/4/3
Table 10: Cell-level effect of supercritical versus published on macro-F1, confirmatory n=8 (dataset, regime) cells (mnist and cifar10 only; see text for the n=9 check with emnist_balanced included), each a mean over that cell’s five seeds and two selection modes. Cells column is favour-supercritical/favour-published/tied; Wilcoxon and sign-test p -values (prose above) are secondary diagnostics, not shown here.
Figure 5: Cell-level macro-F1 difference (supercritical minus published) for each of the nine (dataset, regime) cells, both bases. Circles are the eight confirmatory cells (mnist and cifar10, four regimes each); the diamond is the exploratory emnist_balanced/full cell, added afterward by Amendment 1 and not pooled into the primary statistic. The bottom row shows the confirmatory ( n=8 ) mean difference with its 95% t -interval, the same numbers Table 10 reports. Both intervals straddle zero; most individual cells are exactly tied (difference 0 ) or favour published by a small margin.
Base
supercritical
published
Paired diff.
W
p (raw)
p (Holm)
Brier
0.8136±0.1967
0.8167±0.1955
−0.0031±0.0114
18.0
0.017
0.034
Logarithmic
0.8133±0.1954
0.8162±0.1952
−0.0029±0.0105
18.0
0.017
0.034
Table 11: Non-inferential illustration of pseudoreplication: macro-F1 at supercritical versus published, 90 paired observations per base (dataset × regime × seed × selection mode), mean ± standard deviation across all 90 pairs, Wilcoxon signed-rank, Holm-corrected across the two bases. This table is not evidence; it exists to show what treating pseudoreplicated pairs as independent does to a p -value. The basis for this paper’s conclusions is Table 10 .
λ
value
Base
Macro-F1
ECE
NLL
zero
0
Brier
0.6330±0.0308
0.194±0.031
1.230±0.049
zero
0
Log
0.6263±0.0327
0.176±0.032
1.227±0.047
critical ( λ∗ )
0.4
Brier
0.6397±0.0280
0.208±0.026
1.235±0.049
critical ( λ∗ )
1.609438
Log
0.6409±0.0286
0.209±0.027
1.235±0.050
supercritical
0.400001
Brier
0.6397±0.0280
0.208±0.026
1.235±0.049
supercritical
1.609439
Log
0.6409±0.0286
0.209±0.027
1.235±0.050
Table 12: Macro-F1, ECE, and test NLL across the λ grid, pooled over the four c=10 informative regimes, checkpointing selection. Dispersion is the seed-to-seed standard deviation at a fixed (dataset, regime, λ ) cell, averaged across the four regimes — it reflects seed noise within a cell, not the much larger spread between regimes of very different difficulty. The implemented grid point has no logarithmic row: the mean/sum-form implementation gap is Brier-specific (relation to published rules).
λ
value
Brier macro-F1
Log macro-F1
zero
0
0.6344
0.6294
critical ( λ∗ )
0.4
0.6419
0.6443
supercritical
0.400001
0.6419
0.6443
published
0.9
0.6481
0.6482
implemented
9.0
0.6525
n/a
limit
50.0
0.6527
0.6527
Table 13: Early-stopping selection, macro-F1 only, same four regimes: the shape survives a change of selection rule. The λ column gives the Brier-base value; the log base uses its own value at each named grid point (Table 5 ), omitted here to keep the table compact.
Figure 6: Mean macro-F1 (left axis, solid) and mean ECE (right axis, dashed), each with a shaded band of ±1 seed-to-seed standard deviation averaged across the four pooled regimes (the same dispersion Table 12 reports numerically), across the Brier-base λ grid, checkpointing selection, pooled over the four c=10 informative regimes (mnist/hard, cifar10/subset, cifar10/noisy, cifar10/hard) — λ∗=0.400 and the published penalty 0.900 marked. Both measures rise together across the entire grid; there is no point at which macro-F1 continues improving while ECE holds steady. The bands are wide relative to the rise itself, consistent with the text’s own description of this trajectory as descriptive rather than statistically established. emnist_balanced ( c=47 , a different λ grid) is not pooled into this figure — see the text for its own frontier. Data are derived from the frozen experimental records for the cells trained under the original pre-specified configuration. Unlike Figures 1–4, this figure is experimental, not theoretical, and was built only after the pre-specified sweep completed.
λ
value
Macro-F1
ECE
NLL
zero through subcritical
0 to 0.718
0.8907±0.0022
0.014±0.004
0.309±0.005
published
0.979
0.8900±0.0029
0.015±0.005
0.314±0.013
implemented and limit
46.0 , 200.0
0.8898±0.0030
0.019±0.006
0.318±0.017
Table 14: emnist_balanced/full ( c=47 ), Brier base, checkpointing — its own λ grid, not comparable in absolute value to Tables 12 – 13 . Rows from zero through subcritical are identical (the same checkpoint is selected throughout), so only the distinct values are shown.
w
λ(w)
Superior ( λ(w)>λ∗ )?
0
0
no
0.25
3.33
yes
0.5
10.0
yes
0.75
30.0
yes
0.9
90.0
yes
1.0
∞
yes (limit)
Table 15: The weight-to- λ map λ(w)=wc/(1−w) at c=10 , for every weight tested, against the superiority threshold w∗≈0.038 ( λ∗=0.4 ).
Comparison
Base
Agreement
w=0 vs. λ=0
Brier
90/90 ( 100% )
w=1 vs. λ→∞ (limit)
Brier
82/90 ( 91% )
w=1 vs. λ→∞ (limit)
Logarithmic
81/90 ( 90% )
Table 16: Checkpoint agreement between Mw and the corresponding member of Sλ , at the two endpoints where the identity above applies exactly ( w=0 ) or approximately ( w=1 ), out of 90 (dataset, regime, seed, mode) combinations.
Baseline
Weight
Macro-F1
monitor macro-F1 directly
—
0.7006
monitor error rate directly
—
0.7001
Mw
0.00
0.6845
Mw
0.25
0.6992
Mw
0.50
0.7000
Mw
0.75
0.7001
Table 17: Baseline mean macro-F1, all five informative regimes ( c=10 and c=47 together — weight, unlike λ , is on the same scale for every dataset), checkpointing. " Mw " labels the weighted mixture; despite its original design intent, it is not a no-superiority rule except at w=0 (baselines, above).
Regime
λ=0
supercritical
published
mnist/hard
6.67
6.28
6.19
cifar10/hard
3.78
2.07
2.07
cifar10/noisy
3.05
3.19
1.34
cifar10/subset
1.10
1.10
1.10
emnist_balanced/full
0.89
0.89
1.14
Table 18: Seed-to-seed standard deviation of the selected epoch, Brier base, checkpointing, at three grid points per regime.
Proper scoring rules evaluate probabilistic forecasts and incentivize honest probability reports by rewarding distributions that match observed outcomes.
The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.
This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured.
Benjamin Turtel, Paul Wilczewski, Kris Skotheim +2
Lightning Rod Labs · Technology and Operations Management, INSEAD, Fontainebleau, France · Wharton School and School of Arts & Sciences, University of Pennsylvania
We study how the objective used to train or fine-tune a tabular foundation model affects its distributional predictions. Models such as TabPFN and TabICL provide forecasts of the full conditional distribution, but regression benchmarks including TabArena and TALENT assess them mainly through RMSE and R2. Such assessments reward accurate conditional means while leaving the remaining distributional information unassessed. We therefore examine proper scoring rules as objectives for training and fine-tuning and as criteria for evaluating and ranking these models. Every strictly proper rule favors the true distribution at the population level, however, this shared optimum does not determine how learning proceeds with a finite sample. Our examples illustrate score-dependent preferences between forecasts, and our fine-tuning experiments show differences in predictive performance under different objectives. Experiments that fine-tune TabPFN and TabICL on several datasets reveal that changing the training objective can change performance across evaluation scores, with non-trivial interactions between the two. By connecting established methods from distributional regression, scoring-rule theory, and foundation-model adaptation, we show that selecting the score is part of specifying the model and can measurably alter the distribution it learns.
Jonas Landsgesell, Pascal Knoll, Tizian Wenzel
University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)