Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

Han Chen; Yingrui Li

cs.LGSep 27, 2026

Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

Authors: Han Chen, Yingrui Li

Organizations: Independent Researcher

Abstract

Equivalent probability requests can lead to different decisions even when both reports are valid. We introduce probability contracts, a benchmark that connects exact finite-world posteriors, validated event alignment, interface coherence, and failure-aware decision evaluation. Across four model-interface configurations on 1,000 worlds, Kev has lower aggregate canonical posterior error than Jev but larger complement and coarsening residuals; accuracy ordering varies by stratum. Jev's Event and Choice interfaces change the binary action on 32.8% of valid pairs at defer cost 0.10. Post-hoc analyses show that disagreement certifies only 11-52% of mean binary pair error and does not consistently outperform confidence for selection. An action-region characterization and a standard scoring-rule identity explain averaging's expected Brier guarantee relative to random interface selection, but not a decision-loss guarantee at each cost. The loss contrast takes both signs on a 99-cost grid for every configuration; small penalties where both policies beat deferral have pointwise intervals containing zero. Secondary checks specified before collection include a separate 400-root cohort, where Event/Choice effects remain configuration-dependent. A joint surface-order and answer-ID intervention shifts posttrained probabilities. Both bounded reasoning arms yield no valid probability reports, leaving their probability accuracy undefined. Probability contracts make these distinctions measurable by evaluating event semantics, posterior error, coverage, and decision cost together.

Figures & tables

Relation	Aligned claims	Residual or condition	Eligibility
Complement	$p(A),p(\neg A)$	$\|p(A)+p(\neg A)-1\|$	Exhaustive binary event
Coarsening	$p(A),p(\omega)$	$\|p(A)-\sum_{\omega\in A}p(\omega)\|$	Same partition and information
Event/Choice	$p_{\rm event}(A),p_{\rm choice}(A)$	Absolute difference	Bijection to same event
Intersection	$p(A),p(B),p(A\cap B)$	Distance outside Fréchet bounds	Aligned $A,B$
Product	$p(A\cap B),p(B),p(A\mid B)$	$\|p(A\cap B)-p(B)p(A\mid B)\|$	$q(B)>0$
Permutation	$p(\omega),p_{\pi}(\omega)$	Total variation after inverse map	Bijection; same information

Table 1: Declared probability relations after event alignment. Coarsening compares an event report with the sum of its categories in the canonical report. Fréchet bounds require

\max(0,p(A)+p(B)-1)\leq p(A\cap B)\leq\min(p(A),p(B))

. Continuous residuals are primary; pass thresholds are prespecified summaries.

Figure 1: Canonical posterior error (left), binary Event/Choice disagreement (middle), and canonical decision loss (right); lower is better. Error bars are 95% stratified root-bootstrap intervals. Jev TV uses 990 valid canonical roots; other estimates use 1,000 roots. The outer panels concern the canonical multiclass task; the middle compares reports of the same binary event. The dashed line marks always-defer loss.

Relation	Kev	Base	Posttrained	Jev
Event/Choice	0.0629	0.2138	0.1126	0.1938
Complement	0.5055	0.1155	0.8000	0.0929
Coarsening	0.3554	0.2451	0.2836	0.1751
Intersection bound	0.1553	0.0814	0.0358	0.0200
Product identity	0.1328	0.2682	0.1406	0.0836
Permutation TV	0.1143	0.3224	0.3470	0.0534

Table 2: Mean relation residuals (lower is better within a row). Jev uses 1,000 valid pairs for Event/Choice, complement, and repeat; 990 for coarsening; 999 for intersection; 912 for product; and 976 for permutation. Local support is complete (913 eligible product roots, otherwise 1,000). All intervals and relations are in Appendix K .

	Canonical multiclass task			Binary task
Configuration	Expected loss	Oracle regret	Defer rate	Event/Choice changes
Kev	0.1090 [0.1054, 0.1131]	0.0193 [0.0158, 0.0232]	0.9610 [0.9500, 0.9720]	0.1490 [0.1290, 0.1710]
Base	0.1000 [0.1000, 0.1000]	0.0103 [0.0088, 0.0118]	1.0000 [1.0000, 1.0000]	0.0000 [0.0000, 0.0000]
Posttrained	0.1000 [0.1000, 0.1000]	0.0103 [0.0088, 0.0118]	1.0000 [1.0000, 1.0000]	0.3420 [0.3210, 0.3630]
Jev-1.13.0	0.1336 [0.1263, 0.1414]	0.0438 [0.0366, 0.0515]	0.8160 [0.7920, 0.8380]	0.3280 [0.3020, 0.3540]

Table 3: Decisions at defer cost 0.10: means and 95% root-bootstrap intervals. The first three metrics use the canonical multiclass task; the final column compares separate binary Event/Choice requests. Each uses 1,000 roots per configuration. Canonical loss includes failure-triggered deferral. Always-defer loss is 0.10.

Configuration	Event (1)	Choice (1)	Mean (2)	Agreement (2)	Repeat mean (2)
Kev	0.16695	0.15786	0.15281	0.14186	0.16695
Base	0.10000	0.10000	0.10000	0.10000	0.10000
Posttrained	0.25448	0.23257	0.19055	0.17239	0.25448
Jev	0.09654	0.17052	0.09890	0.09654	0.09645

Table 4: Post-hoc binary policy loss at

c=0.10

(all 1,000 roots per configuration; lower is better). Parentheses in headers give requests per decision. Agreement means common action or defer. Always-defer loss is 0.10000 and binary oracle loss is 0.08132. Intervals and full comparisons are in Appendix P .

Figure 2: Mean minus Random interface expected binary loss at the 99 evaluated costs (post hoc). Positive values favor Random interface. Lines connect grid estimates; Appendix S supplies uncertainty and exact breakpoint records.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Defer cost	Action disagreement	95% interval	Loss difference	95% interval
0.02	0.0535	[0.0495, 0.0575]	$0.0240$	[ $0.0217$ , $0.0263$ ]
0.05	0.1980	[0.1865, 0.2095]	$-0.0083$	[ $-0.0147$ , $-0.0020$ ]
0.10	0.6085	[0.5980, 0.6195]	$-0.1917$	[ $-0.2006$ , $-0.1827$ ]
0.20	0.6780	[0.6660, 0.6905]	$-0.2020$	[ $-0.2093$ , $-0.1943$ ]
0.40	0.2705	[0.2615, 0.2795]	$-0.0341$	[ $-0.0411$ , $-0.0270$ ]

Appendix

Table 5: Policy consequences of the joint surface-order and answer-ID intervention. Both event

A

and its complement are retained (2,000 paired reports on 1,000 roots); the two variant differences are averaged within root before stratified resampling. Loss differences are recoded minus historical, and action disagreement compares the two report-based controllers. Intervals are pointwise 95% summaries. All policies use the original reports, with action selection independent of evaluator truth.

Configuration	Valid	Accuracy	TV	Exp. Brier	Gap	Complement	Repeat
Kev (FP32)	2,000/2,000	0.460	0.3426	0.6167	0.0772	0.5217	0.0000
Base (FP32)	2,000/2,000	0.365	0.4326	0.7185	0.2109	0.1186	0.0000
Posttrained (FP32)	2,000/2,000	0.420	0.4001	0.6786	0.1156	0.7929	0.0000
Reasoning (BF16)	0/2,000	–	–	–	–	–	–

Appendix

Table 6: Local results on the 400-root headroom cohort. Valid counts requests across variants. Accuracy uses the sampled label; accuracy, TV, and expected Brier each use 400 valid canonical reports per non-reasoning configuration. Gap denotes mean absolute Event/Choice difference; all three relation residuals use 400 valid pairs per non-reasoning configuration. Dashes indicate undefined probability metrics for Reasoning: all 2,000 requests reached the reasoning limit.

Figure 3: The five prespecified costs, connected to show the sampled trend. Expected decision loss and deferral are shown separately; the dashed loss line is always defer. The primary cost is 0.10. Discussed in Section 6 .

Variant	Intended	Eligible
Canonical	1,000	1,000
Event	1,000	1,000
Complement $\neg A$	1,000	1,000
Query of event $B$	1,000	1,000
Intersection	1,000	1,000
Conditional	1,000	913

Appendix

Table 7: Per-configuration main requests.

Configuration	Valid canonical roots	Posterior TV	Expected Brier
Kev	1,000	0.2350 [0.2278, 0.2423]	0.6333 [0.6229, 0.6436]
Base	1,000	0.3740 [0.3655, 0.3824]	0.7474 [0.7379, 0.7571]
Posttrained	1,000	0.3202 [0.3124, 0.3280]	0.6938 [0.6831, 0.7048]
Jev-1.13.0	990	0.2677 [0.2599, 0.2754]	0.6560 [0.6415, 0.6705]

Appendix

Table 8: Canonical posterior accuracy: eight-stratum means with 95% paired root-bootstrap intervals. Scores use valid canonical roots; all 1,000 intended roots enter decision loss. Full response counts and realized Brier scores appear in Appendix D .

Configuration	Realized Brier
Kev	0.6359 [0.6160, 0.6571]
Base	0.7530 [0.7332, 0.7739]
Posttrained	0.6918 [0.6696, 0.7137]
Jev-1.13.0	0.6469 [0.6094, 0.6851]

Appendix

Table 9: Realized canonical Brier: mean [95% interval].

Configuration	Native validity	Exact- $q$ gap	Operational criterion
Kev	1.0000	0.0618	No
Base	1.0000	0.1441	No
Posttrained	1.0000	0.0877	No
Jev-1.13.0	0.9946	0.1952	No

Appendix

Table 10: Symmetric all-configuration check. Exact-

q

gap is a 15-bin top-class calibration summary against evaluator posterior mass.

Relation	Mean [95% interval]	Valid roots	Eligible roots
Complement	0.5055 [0.4971, 0.5138]	1000	1000
Event/Choice interface	0.0629 [0.0610, 0.0648]	1000	1000
Coarsening	0.3554 [0.3489, 0.3620]	1000	1000
Intersection bounds	0.1553 [0.1507, 0.1598]	1000	1000
Product	0.1328 [0.1300, 0.1356]	913	913
Exact repeat	0.0000 [0.0000, 0.0000]	1000	1000

Appendix

Table 11: All continuous relation residuals for Kev. Estimates and intervals preserve each relation’s paired-valid support.

Relation	Mean [95% interval]	Valid roots	Eligible roots
Complement	0.1155 [0.1133, 0.1177]	1000	1000
Event/Choice interface	0.2138 [0.2111, 0.2165]	1000	1000
Coarsening	0.2451 [0.2420, 0.2481]	1000	1000
Intersection bounds	0.0814 [0.0807, 0.0821]	1000	1000
Product	0.2682 [0.2673, 0.2691]	913	913
Exact repeat	0.0000 [0.0000, 0.0000]	1000	1000

Appendix

Table 12: All continuous relation residuals for Base. Estimates and intervals preserve each relation’s paired-valid support.

Relation	Mean [95% interval]	Valid roots	Eligible roots
Complement	0.8000 [0.7982, 0.8017]	1000	1000
Event/Choice interface	0.1126 [0.1085, 0.1168]	1000	1000
Coarsening	0.2836 [0.2776, 0.2896]	1000	1000
Intersection bounds	0.0358 [0.0342, 0.0375]	1000	1000
Product	0.1406 [0.1389, 0.1424]	913	913
Exact repeat	0.0000 [0.0000, 0.0000]	1000	1000

Appendix

Table 13: All continuous relation residuals for Posttrained. Estimates and intervals preserve each relation’s paired-valid support.

Relation	Mean [95% interval]	Valid roots	Eligible roots
Complement	0.0929 [0.0890, 0.0967]	1000	1000
Event/Choice interface	0.1938 [0.1873, 0.2002]	1000	1000
Coarsening	0.1751 [0.1690, 0.1813]	990	1000
Intersection bounds	0.0200 [0.0173, 0.0227]	999	1000
Product	0.0836 [0.0804, 0.0868]	912	913
Exact repeat	0.0188 [0.0176, 0.0200]	1000	1000

Appendix

Table 14: All continuous relation residuals for Jev-1.13.0. Estimates and intervals preserve each relation’s paired-valid support.

View	Question
Eligible	Was the semantic transformation defined?
Native valid	Did the returned object satisfy its interface contract?
Paired valid	Are all vectors needed for this contrast scoreable?
Intended pipeline	What loss results when failure triggers defer?

Appendix

Table 15: Four accounting views retained for every configuration.

Stratum	Kev	Base	Posttrained	Jev
Finite population / joined tables	0.2696	0.4061	0.2381	0.2429
Finite population / nested records	0.1866	0.3905	0.3408	0.1858
Joint attributes / joined tables	0.1764	0.3097	0.2468	0.4017
Joint attributes / nested records	0.1868	0.2937	0.3489	0.3831
Missing rule / joined tables	0.4545	0.4111	0.3430	0.1931
Missing rule / nested records	0.1598	0.3614	0.3516	0.1272

Appendix

Table 16: Canonical posterior TV by prespecified generator–encoding stratum. Every specified stratum is shown, including reversals of aggregate ordering.

Configuration	Mean	Minimum	Maximum	$s>1$
Kev	1.5055	1.0485	1.8153	1000
Base	1.1103	0.9344	1.3733	861
Posttrained	1.8000	1.5743	1.9150	1000
Jev	1.0641	0.6700	1.3900	786

Appendix

Table 17: Signed complement sums on 1,000 pairs per configuration. Post-hoc summaries; all configurations retained.

Configuration	Valid requests	Canonical TV ( $n$ )	Complement gap ( $n$ )	Loss	Defer
Mistral likelihood	2385/2385	0.4799 (200)	0.6469 (200)	0.1734	0.8900
Mistral JSON	1454/2385	0.3937 (59)	0.4848 (139)	0.1119	0.9650
OLMo likelihood	2385/2385	0.4261 (200)	0.2048 (200)	0.1000	1.0000
OLMo JSON	1248/2385	0.4220 (60)	0.2014 (103)	0.1142	0.9700

Appendix

Table 18: Four new configurations on the same 200-world subset. Validity uses all eligible requests; parentheses give valid canonical or paired-valid roots. Loss and defer rate retain all 200 roots at

c=0.10

. Rejected outputs defer. Valid-only means average nonempty strata; full support and pointwise intervals are in Appendix N .

Reason	Mistral	OLMo
Invalid JSON	528	4
Wrong key support	0	536
Nonnumeric probability	21	0
Trailing content	8	0
Out-of-range probability	62	1
Simplex sum mismatch	312	596

Appendix

Table 19: Strict generated-JSON rejection reasons across all 2,385 main requests per model. Categories are mutually exclusive parser outcomes; none is repaired. These are protocol-validity failures, distinct from relation residuals on valid reports.

Configuration	$n$ valid	TV	Expected Brier	Expected loss
Mistral likelihood	200	0.4799 [0.4517, 0.5099]	0.9696 [0.9275, 1.0132]	0.1734 [0.1531, 0.1942]
Mistral JSON	59	0.3937 [0.3506, 0.4358]	0.8126 [0.7548, 0.8725]	0.1119 [0.1001, 0.1271]
OLMo likelihood	200	0.4261 [0.3955, 0.4583]	0.8566 [0.8224, 0.8913]	0.1000 [0.1000, 0.1000]
OLMo JSON	60	0.4220 [0.3767, 0.4666]	0.8224 [0.7598, 0.8884]	0.1142 [0.1020, 0.1299]
Kev (historical)	200	0.2349 [0.2179, 0.2522]	0.6239 [0.6028, 0.6445]	0.1079 [0.1016, 0.1149]
Base (historical)	200	0.3785 [0.3601, 0.3967]	0.7474 [0.7250, 0.7698]	0.1000 [0.1000, 0.1000]

Appendix

Table 20: Canonical subset results with 95% pointwise root-bootstrap intervals. Probability scores use the stated valid support; expected decision loss uses all 200 roots at

c=0.10

. Historical rows reuse earlier responses on exactly this subset and are not a controlled architecture comparison.

Relation	$n$ likelihood	Likelihood residual	$n$ JSON	JSON residual
Event/Choice	200	0.0000 [0.0000, 0.0000]	140	0.0000 [0.0000, 0.0000]
Complement	200	0.6469 [0.6340, 0.6600]	139	0.4848 [0.4423, 0.5269]
Coarsening	200	0.2882 [0.2730, 0.3034]	51	0.2055 [0.1664, 0.2448]
Intersection bound	200	0.1091 [0.1052, 0.1131]	124	0.2184 [0.1904, 0.2490]
Product	185	0.1575 [0.1527, 0.1625]	146	0.3279 [0.2957, 0.3626]
Permutation	200	0.6167 [0.5891, 0.6432]	39	0.4039 [0.3559, 0.4553]

Appendix

Table 21: Mistral relation residuals with pointwise intervals. Each readout has 200 intended roots, except product (185 eligible).

n

counts complete valid constituents. Missing strata are excluded only from valid-only aggregates; the machine-readable support inventory identifies every exclusion.

Relation	$n$ likelihood	Likelihood residual	$n$ JSON	JSON residual
Event/Choice	200	0.0000 [0.0000, 0.0000]	196	0.0000 [0.0000, 0.0000]
Complement	200	0.2048 [0.2012, 0.2084]	103	0.2014 [0.1742, 0.2297]
Coarsening	200	0.4655 [0.4599, 0.4711]	58	0.2456 [0.1977, 0.2937]
Intersection bound	200	0.0062 [0.0053, 0.0071]	46	0.2206 [0.1730, 0.2713]
Product	185	0.0984 [0.0958, 0.1010]	45	0.3613 [0.3201, 0.4026]
Permutation	200	0.4890 [0.4633, 0.5152]	34	0.4036 [0.3343, 0.4757]

Appendix

Table 22: OLMo relation residuals with pointwise intervals. Each readout has 200 intended roots, except product (185 eligible).

n

counts complete valid constituents. Missing strata are excluded only from valid-only aggregates; the machine-readable support inventory identifies every exclusion.

Configuration	$c=.02$	$c=.05$	$c=.10$	$c=.20$	$c=.40$
Mistral likelihood	0.0200	0.0581	0.1734	0.3124	0.5635
Mistral JSON	0.0200	0.0523	0.1119	0.2140	0.4304
OLMo likelihood	0.0200	0.0500	0.1000	0.2000	0.4304
OLMo JSON	0.0299	0.0640	0.1142	0.2177	0.4195
Kev (historical)	0.0200	0.0500	0.1079	0.1999	0.3913
Base (historical)	0.0200	0.0500	0.1000	0.2000	0.3844

Appendix

Table 23: Expected canonical decision loss for the complete fixed cost grid, all 200 roots per configuration. Rejected and unavailable outputs defer. Always-defer loss equals the column cost.

Contrast	$n$ TV	TV difference	Expected-loss difference
Mistral likelihood – Mistral JSON	59	0.1145 [0.0454, 0.1855]	0.0615 [0.0342, 0.0871]
Mistral likelihood – OLMo likelihood	200	0.0538 [0.0331, 0.0750]	0.0734 [0.0531, 0.0942]
Mistral likelihood – OLMo JSON	60	0.0790 [0.0015, 0.1582]	0.0591 [0.0368, 0.0816]
Mistral JSON – OLMo likelihood	59	-0.0780 [-0.1505, -0.0077]	0.0119 [0.0001, 0.0271]
Mistral JSON – OLMo JSON	21	-0.1047 [-0.1629, -0.0506]	-0.0024 [-0.0226, 0.0179]
OLMo likelihood – OLMo JSON	60	-0.0164 [-0.0902, 0.0558]	-0.0142 [-0.0299, -0.0020]

Appendix

Table 24: All six planned configuration contrasts (left minus right). TV uses paired-valid canonical roots; loss uses all 200 roots at

c=0.10

. Intervals are pointwise, without multiplicity-adjusted superiority claims. The full result file also includes Brier and the other fixed costs.

Configuration and estimand	F/T	F/R	J/T	J/R	M/T	M/R	N/T	N/R
Mistral likelihood, canonical	25	25	25	25	25	25	25	25
Mistral likelihood, Event/Choice	25	25	25	25	25	25	25	25
Mistral JSON, canonical	8	1	8	2	15	16	8	1
Mistral JSON, Event/Choice	25	12	18	7	24	24	19	11
OLMo likelihood, canonical	25	25	25	25	25	25	25	25
OLMo likelihood, Event/Choice	25	25	25	25	25	25	25	25

Appendix

Table 25: Valid-root support by the eight fixed strata (25 intended roots each): finite population (F), joint attributes (J), missing rule (M), and noisy sensor (N), each in joined-table (T) and nested-record (R) form. Zero-support cells are excluded from valid-only means; all cells remain in decision loss. Canonical and Event/Choice supports need not coincide.

Configuration	Pair absolute error	Disagreement lower bound	Error beyond bound
Kev	0.29397 [0.28537, 0.30287]	0.03143 [0.03048, 0.03240]	0.26254 [0.25384, 0.27140]
Base	0.25692 [0.24884, 0.26514]	0.10689 [0.10553, 0.10816]	0.15004 [0.14149, 0.15864]
Posttrained	0.37758 [0.36562, 0.38971]	0.05632 [0.05423, 0.05836]	0.32126 [0.30941, 0.33318]
Jev	0.18567 [0.18023, 0.19105]	0.09688 [0.09362, 0.10010]	0.08879 [0.08376, 0.09393]

Appendix

Table 26: Post-hoc absolute-error decomposition for Event/Choice: mean and pointwise 95% stratified root-bootstrap intervals. The disagreement lower bound plus error beyond the bound equals mean pair error before rounding.

Configuration	Mean minus Random interface	Agreement minus mean
Kev	-0.00959 [-0.01347, -0.00581]	-0.01095 [-0.01503, -0.00733]
Base	0.00000 [0.00000, 0.00000]	0.00000 [0.00000, 0.00000]
Posttrained	-0.05297 [-0.05964, -0.04634]	-0.01816 [-0.02399, -0.01291]
Jev	-0.03463 [-0.03886, -0.03049]	-0.00236 [-0.00411, -0.00077]

Appendix

Table 27: Post-hoc primary reporting contrasts at

c=0.10

: expected loss difference with pointwise 95% intervals. Negative favors the first policy. Random interface selects Event or Choice with equal probability. No multiplicity-adjusted significance claim is made.

Configuration	$c$	Event	Choice	Mean	Agreement	Repeat mean	Repeat agreement	Oracle
Kev	0.02	0.0200	0.0200	0.0200	0.0200	0.0200	0.0200	0.0165
Kev	0.05	0.0512	0.0516	0.0506	0.0501	0.0512	0.0512	0.0411
Kev	0.10	0.1669	0.1579	0.1528	0.1419	0.1669	0.1669	0.0813
Kev	0.20	0.3128	0.3052	0.3122	0.2885	0.3128	0.3128	0.1571
Kev	0.40	0.3988	0.3986	0.3967	0.3950	0.3988	0.3988	0.2689
Base	0.02	0.0200	0.0200	0.0200	0.0200	0.0200	0.0200	0.0165

Appendix

Table 28: Post-hoc binary expected loss across all fixed costs. Every configuration has 1,000 intended roots and complete inputs. Event/Choice use one query; mean/agreement and repeat controls use two. Always defer has loss

c

. Oracle is evaluator-only. Full pointwise intervals and coverage remain in the machine-readable results.

Figure 4: Post-hoc Event/Choice error decomposition on all 1,000 pairs per configuration. The disagreement lower bound and the remaining component sum to the corresponding mean pair error. Ratios describe these reports and worlds; they are not causal attributions or a universal calibration bound.

Figure 5: Post-hoc fixed-policy loss and coverage at the primary defer cost

c=0.10

. Each configuration is shown separately. Always defer is a cost reference; the oracle uses evaluator truth. The policy/contrast tables supply pointwise paired intervals.

Original actions	Midpoint action	$\Delta_{c}$
Same action	Same action	$0$
No, defer	No	$(q-c)/2$
No, defer	Defer	$(c-q)/2$
Defer, yes	Defer	$(c+q-1)/2$
Defer, yes	Yes	$(1-q-c)/2$
No, yes	No	$q-1/2$

Appendix

Table 29: Exact per-world penalty of the mean policy versus uniformly selecting one of the two reports. The first column is an unordered action pair; rows apply when the stated midpoint action is feasible.

Figure 6: Post-hoc expected binary reject-option loss over the symmetric defer cost. Panels use common axes. Lines are descriptive point estimates; exact breakpoints and pointwise bootstrap intervals are supplied in the artifact. Lines connect the evaluated grid points and need not reproduce intervening jumps; the exact breakpoint records, rather than these lines, determine between-grid behavior.

Figure 7: Post-hoc expected inclusive elementary score over threshold

\theta

. Panels use common axes. This Murphy diagram is distinct from Figure 6 ; it supports only the corresponding binary proper-score-mixture interpretation. Connecting lines are a grid visualization, not the exact between-grid curve.

Configuration	Lower	Equal	Higher	Both below
Jev	82	1	16	16
Kev	62	5	32	7
Base	47	36	16	0
Posttrained	47	1	51	2

Appendix

Table 30: Signs of mean-minus-Random interface expected loss on the post-hoc 99-cost grid. Counts concern grid locations, not independent trials or shares of the cost continuum. “Both below” counts positive contrasts for which both policies beat always defer.

Configuration	Disagreement	Confidence	Random
Jev	0.17311	0.17943	0.22314
Kev	0.25827	0.23729	0.25416
Base	0.29737	0.29122	0.29496
Posttrained	0.29110	0.27650	0.29496

Appendix

Table 31: Post-hoc binary pipeline loss at 50% retained coverage and

c=0.10

. Every strategy uses the same mean-report class predictor on retained worlds; unretained worlds defer. Entries are point estimates; the artifact supplies paired uncertainty summaries.

Figure 8: Post-hoc binary pipeline loss at

c=0.10

versus retained coverage. Panels use common axes. Disagreement, confidence, and random retention share the same fixed class predictor and candidate population; the oracle-excess ordering uses evaluator truth and is nondeployable. It ranks class excess risk and is not an optimal envelope for pipeline loss. At full coverage this is a binary class predictor without the original reject option.

Runtime stage	Mode	Acquired	Valid	Capped	Interrupted
Original	Thinking	6	3	2	1
Original	Non-thinking	6	6	0	0
Optimized single replica	Thinking	5	4	0	1
Optimized single replica	Non-thinking	4	4	0	0
Four optimized replicas	Thinking	69	64	5	0
Four optimized replicas	Non-thinking	69	69	0	0

Appendix

Table 32: Operational diagnostic counts by runtime stage and mode. Valid and capped requests have terminal records; interrupted requests do not. Counts describe the acquired requests in each stage, not estimates of accuracy or a pooled completion rate. One further planned non-thinking complement request was never acquired.

Request	$n$	$\Delta$ TV	$\Delta$ Brier	$\Delta L_{0.10}$	$\Delta L_{0.40}$
Canonical	12	-0.0789	-0.0581	+0.2902	-0.1969
Event	15	-0.1331	-0.1517	+0.1068	-0.3008
Choice	12	-0.2735	-0.4217	+0.0529	-0.3893
Complement	13	-0.0423	-0.0287	+0.1562	-0.1480
Repeat	12	-0.0968	-0.1138	+0.1313	-0.2911

Appendix

Table 33: Exploratory matched thinking-minus-non-thinking differences in the final runtime stage. Each row uses the same

n

roots with valid reports in both modes. Negative values favor thinking. TV and expected Brier concern the stated request partition; binary Brier includes both coordinates. These are descriptive means from at most 16 worlds, with dependent request types and no confidence intervals. All declared costs, earlier stages, root IDs, and failure-fallback scores are supplied in the artifact.

Explore similar work

Sep 27, 2026cs.CL

Do System One Decisions Add Up? A Study of Probabilistic Coherence

A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.

Saman Sarker Joy

Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia

Confidence Calibration Decisions Motion Coherence Joint Search Intermediate Checkpoints

Ask PDF

Sep 29, 2026cs.AI

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Large language models (LLMs) are increasingly proposed as decision assistants who must reason probabilistically from available evidence under explicit decision costs. We propose a decision-theoretic framework that decomposes LLMs' decision loss into two components: forming accurate beliefs from provided evidence and translating those beliefs into actions that optimize a provided utility function. Using a synthetic benchmark with known ground truth, we apply the decomposition to characterize probabilistic reasoning in frontier and open-sourced models. We further evaluate whether RL interventions targeting beliefs, decisions, or both improve these components across three domains, whether improvements transfer across components and elicitation formats, and whether decision performance can improve without improvement in belief formation. We find that targeting one component of probabilistic reasoning redistributes decision loss, improving the target without necessarily transferring to others, and that jointly targeting belief formation and decision-making improves both but hinges on matched formats between training and evaluation.

Huaman Sun, Dingcheng Wang, Jason Hartline +1

Department of Computer Science Northwestern University Evanston, IL 60201, USA

LLM Reasoning Strategies Probabilistic Model Elicitation Diagnosis

Ask PDF

Jun 5, 2026cs.CL

How reliable are LLMs when it comes to playing dice?

We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when canonical formulations are replaced by disguised variants. Embedding misleading suggestions in the prompt reduces performance by up to 34%, with no model proving immune. Taken together, the reported findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems.

Luca Avena, Gianmarco Bet, Bernardo Busoni

Università degli Studi di Firenze Dipartimento di Matematica e Informatica ‘Ulisse Dini’

LLM Reasoning Strategies Probabilistic Model Chain-of-Thought Reasoning

Ask PDF