Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.
Figures & tables
Figure 1: Direct evaluation of model probabilities against exact class probabilities using KL divergence. We compare 16 models given 25 labeled memos per test case (means over three seeds). Filled markers denote decision models; open markers denote generative models. Red lines connect the observed Pareto frontiers. Lower price and KL are better.
Figure 2: How a world generates a case. Both g and ϕ are deterministic. ML models read Xvis , while language models and agents read only the memo T .
Figure 3Figure 4
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
In
Out
Source
Decision models
Jev 1.13 Noul
0.042
0.00
TypeSafe AI: Jev announcement
GPT-6 Luna Decisions
0.10
0.00
OpenAI: Decisions pricing
Liquid d1
0.04
0.00
Liquid AI: d1 announcement
Perplexity Decider V1.1 27B
0.02
0.00
OpenRouter: Decisions catalog
Clef
0.24
0.00
Cloudflare: Workers AI pricing
Appendix
Table 1: Published list prices used in Fig. 1 , in USD per million tokens. Source dates: 2026-10-09 for Claude and Perplexity; 2026-10-07 for decision models other than Jev and Perplexity; 2026-09-21 for the remaining models.
Dataset
Setting
Visible information
Bayes error
Entropy (bits)
Peer review
Low
Contribution type and all eight factors
0.034
0.165
Medium
Low − novelty
0.091
0.280
High
Medium − technical validity
0.109
0.361
Covert transfer
Low
Report type, identifier; tag’s broad range, narrow range, letter
0.019
0.119
Medium
Low − exact tag letter
0.074
0.317
High
Medium − narrow range
0.145
0.490
Appendix
Table 2: Bayes error and conditional label entropy of each setting. Each subtraction starts from the named preceding setting.
World
Factor (unit)
Rendered values for levels 0 / 1 / 2
Peer review
Novelty ∗
incremental / substantial / major
Problem significance
narrow / moderate / broad
Technical validity ∗∗
serious flaw / limited concern / sound
Supporting evidence (rating out of 5)
1.0–2.4 / 2.5–3.4 / 3.5–5.0
Comparison quality
inadequate / partly controlled / well controlled
Robustness
fragile / mixed / stable
Appendix
Table 3: Substantive factors and their rendered values for latent levels 0/1/2 . Numerical ranges give inclusive grid endpoints. ∗ Hidden at Medium and High; ∗∗ only at High.
Nov.
Sig.
Val.
Evid.
Comp.
Rob.
Repr.
Clar.
Theoretical
1.7
1.2
2.0
1.8
0.7
0.7
0.8
1.1
Empirical
1.0
1.0
1.5
2.5
1.3
1.1
0.9
0.7
Applied
0.9
1.8
1.3
2.5
0.9
1.0
0.8
0.8
Appendix
Table 4: Peer-review factor weights by contribution type. Each row sums to 10.
Peer review
Covert transfer
Credit risk
Authors (1–20)
Audit month (Mar., Sep.)
Employees (20–5,000)
Manuscript pages (4–30)
Report revision (1, 2)
Operating sites (1–30)
Figures (0–20)
Attachments (0–20)
Departments (1–20)
Tables (0–15)
Logging region (East, West)
Reporting month (Mar., Sep.)
References (5–150)
Export version (1, 2)
Fiscal year end (Jun., Dec.)
Appendix pages (0–60)
Retention tier (standard, extended)
Report pages (4–80)
Appendix
Table 5: Irrelevant fields. Numerical ranges are inclusive, categorical pairs are equiprobable, and submission month is uniform over twelve months. All fields are mutually independent.
Peer review
Covert transfer
Credit risk
Question
will it be accepted?
was the tag encoded?
will the company default?
Always visible
contribution type
report type, identifier
sector, business cycle
Latent factors
8 factors, residual
tag, procedure, letters
8 factors, residual
Irrelevant fields
8
8
8
Positive-class prior
0.241 (accept)
0.5 (encoding)
0.211 (default)
Bayes error range
0.034–0.109
0.019–0.145
0.026–0.167
Appendix
Table 6: The three ProbabilityBench worlds.
Setting
s=0
0.5
1
1.5
2
Peer-review world
Low
0.000
0.017
0.034
0.051
0.067
Medium
0.075
0.083
0.091
0.098
0.106
High
0.092
0.100
0.109
0.117
0.125
Credit risk world
Low
0.000
0.013
0.026
0.039
0.052
Appendix
Table 7: Exact Bayes error as the residual’s nonzero mass varies, enumerated without fitting models. Scale s multiplies each nonzero residual probability, assigning the remaining mass to zero. The benchmark uses s=1 . Thus P(R=0)=0.28s in review and 0.24s in credit risk.
Figure 11: Expected accuracy (Eq. 1 ) of the predictions in Fig. 4 , in peer review (a), covert transfer (b), and credit risk (c), using 50 training examples per method and three seeds (mean ± s.d.). Dashed lines show mean exact Bayes accuracy on the same test inputs, which expected accuracy cannot exceed, unlike realized accuracy. The y-axis ranges differ across panels.