Do System One Decisions Add Up? A Study of Probabilistic Coherence

Saman Sarker Joy

cs.CLSep 27, 2026

Do System One Decisions Add Up? A Study of Probabilistic Coherence

Authors: Saman Sarker Joy

Organizations: Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia

Abstract

A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.

Figures & tables

Dataset	Examples	Parents	Fine labels	Represented	Selection
TREC	500	6	50	42	Entire test split
CLINC150	1,000	10	150	150	Stratified in-scope test
MASSIVE en-US	1,000	18	60	59	Capacity-aware test sample

Table 1: Fixed evaluation design. Represented labels are fine labels with at least one selected gold example; all declared labels remain candidates. Counts describe the fixed evaluation sample.

System	Identity	Execution	Qualification
Jev	jev-1.13.0	TypeSafe API; precision undisclosed	Proprietary training and serving details
Laya	Pinned English root checkpoint	Local FP32; runtime 0.3.20	Expanded context; released option-count calibration

Table 2: Evaluated configurations. Neither system is fitted to the study examples. Architecture descriptions rely on release documentation; hosted internals remain unverified.

Dataset	Model	F acc.	F F1	H acc.	H F1	M acc.	M F1
TREC	Jev	72.2	48.6	57.0	35.7	60.4	37.4
TREC	Laya	28.8	24.7	33.4	31.7	34.4	32.9
CLINC150	Jev	91.0	90.4	64.6	62.3	68.1	66.1
CLINC150	Laya	23.1	28.3	33.9	33.9	44.4	44.6
MASSIVE	Jev	83.3	81.1	65.9	65.1	66.9	66.0
MASSIVE	Laya	23.8	24.8	32.9	30.4	33.0	30.3

Table 3: Fine-label performance (percent). Macro F1 includes every declared class, including absent test classes. F, H, and M denote flat, hard routing, and marginalization.

Dataset	Model	Flat	Marginal	Change
TREC	Jev	72.2 [69.2, 75.0]	60.4 [57.0, 63.8]	-11.8 [-15.2, -8.4]
TREC	Laya	28.8 [25.6, 32.0]	34.4 [31.2, 37.8]	+5.6 [+1.4, +9.8]
CLINC150	Jev	91.0 [89.6, 92.3]	68.1 [66.1, 70.0]	-22.9 [-24.9, -20.9]
CLINC150	Laya	23.1 [20.9, 25.4]	44.4 [42.1, 46.7]	+21.3 [+18.0, +24.5]
MASSIVE	Jev	83.3 [81.2, 85.3]	66.9 [64.5, 69.3]	-16.4 [-18.6, -14.2]
MASSIVE	Laya	23.8 [21.5, 26.2]	33.0 [30.8, 35.2]	+9.2 [+6.3, +12.1]

Table 4: Accuracy and paired changes with 95% stratified percentile-bootstrap intervals (10,000 replicates). Accuracies are percentages; changes are percentage points.

Dataset	Model	Coarse B	Coarse E	Flat B	Flat E	Marg. B	Marg. E
TREC	Jev	0.347	0.082	0.401	0.058	0.560	0.079
TREC	Laya	0.175	0.033	0.928	0.203	0.742	0.169
CLINC150	Jev	0.463	0.137	0.136	0.020	0.440	0.048
CLINC150	Laya	0.717	0.105	0.913	0.160	0.714	0.080
MASSIVE	Jev	0.380	0.075	0.245	0.036	0.455	0.061
MASSIVE	Laya	0.753	0.258	0.884	0.046	0.841	0.124

Table 5: Brier score (B) and 15-bin top-label ECE (E), lower is better. Brier uses the sum of classwise squared errors. Coarse scores concern a different label space.

Dataset	Model	CFCE	Parent TV	Gold drift	JSD	Agree (%)
TREC	Jev	0.073	0.219	0.209	0.142	70.6
TREC	Laya	0.141	0.424	0.223	0.438	36.0
CLINC150	Jev	0.070	0.349	0.314	0.281	69.3
CLINC150	Laya	0.117	0.587	0.297	0.591	12.3
MASSIVE	Jev	0.029	0.257	0.228	0.188	75.7
MASSIVE	Laya	0.077	0.689	0.219	0.572	18.4

Table 6: Mean coherence discrepancies and flat–marginal label agreement. TV, drift, and JSD range from zero to one; JSD uses bits. CFCE and TV are algebraically redundant; compare TV across datasets.

Dataset	Model	Flat TV	Flat flip (%)	Marg. TV	Marg. flip (%)
TREC	Jev	0.168	20.0	0.140	17.5
TREC	Laya	0.491	62.5	0.221	39.2
CLINC150	Jev	0.062	5.8	0.150	19.2
CLINC150	Laya	0.554	61.7	0.368	41.7
MASSIVE	Jev	0.069	5.8	0.097	14.2
MASSIVE	Laya	0.450	76.7	0.247	25.8

Table 7: Option-order audit: mean TV and label-flip percentage against the main order. Each row uses 30 examples with four permutations (120 paired comparisons sharing 30 inputs).

Dataset	Model	Both	Lost	Gained	Neither	Wrong parent	Within
TREC	Jev	274	87	28	111	117	98
CLINC150	Jev	660	250	21	69	317	37
MASSIVE	Jev	651	182	18	149	271	70
TREC	Laya	92	52	80	276	54	279
CLINC150	Laya	107	124	337	432	596	65
MASSIVE	Laya	132	106	198	564	523	148

Table 8: Paired error accounting (counts). Both/lost/gained/neither refer to correctness under flat and marginal predictions and sum to N. Wrong parent plus within-parent errors plus hard-correct cases separately sum to N.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	TV	Low	High	JSD	Low	High
TREC	Jev	0.219	0.208	0.230	0.142	0.134	0.149
TREC	Laya	0.424	0.403	0.446	0.438	0.421	0.456
CLINC150	Jev	0.349	0.337	0.361	0.281	0.271	0.292
CLINC150	Laya	0.587	0.576	0.598	0.591	0.581	0.601
MASSIVE	Jev	0.257	0.244	0.269	0.188	0.179	0.199
MASSIVE	Laya	0.689	0.677	0.702	0.572	0.561	0.583

Appendix

Table 9: Mean parent TV and JSD with 95% stratified bootstrap intervals.

Dataset	Model	Brier change	ECE change
TREC	Jev	+0.159 [+0.135, +0.184]	+0.021 [-0.018, +0.057]
TREC	Laya	-0.186 [-0.228, -0.144]	-0.033 [-0.076, +0.010]
CLINC150	Jev	+0.303 [+0.284, +0.323]	+0.028 [+0.010, +0.048]
CLINC150	Laya	-0.199 [-0.228, -0.170]	-0.080 [-0.101, -0.042]
MASSIVE	Jev	+0.210 [+0.189, +0.231]	+0.026 [+0.005, +0.049]
MASSIVE	Laya	-0.043 [-0.069, -0.017]	+0.078 [+0.044, +0.096]

Appendix

Table 10: Paired marginal-minus-flat changes in Brier and ECE with 95% bootstrap intervals. Negative changes indicate lower loss or calibration error.

Dataset	Flat TV	Flat flip (%)	Marg. TV	Marg. flip (%)
TREC	0.042	1.7	0.040	3.3
CLINC150	0.017	0.0	0.041	8.3
MASSIVE	0.016	0.0	0.030	8.3

Appendix

Table 11: Jev identical-request audit: 30 examples per dataset, each repeated twice (60 paired comparisons). These are descriptive summaries of rounded outputs.

Dataset	Model	High-conf. N	Errors	Acc. (%)	JSD correct	JSD wrong
TREC	Jev	169	13	92.3	0.146	0.130
CLINC150	Jev	849	28	96.7	0.272	0.374
MASSIVE	Jev	735	43	94.1	0.178	0.242
TREC	Laya	127	65	48.8	0.292	0.498
CLINC150	Laya	145	39	73.1	0.496	0.619
MASSIVE	Laya	89	27	69.7	0.464	0.606

Appendix

Table 12: Exploratory flat-confidence and correctness subgroups. High confidence means maximum flat probability at least 0.8. JSD columns condition on flat correctness; subgroup membership is not randomized.

Dataset	Model	Flat ties	Flat range	Marg. ties	Marg. range
TREC	Jev	6	71.8–72.6	3	60.2–60.8
CLINC150	Jev	1	90.9–91.0	0	68.1–68.1
MASSIVE	Jev	0	83.3–83.3	1	66.9–67.0
TREC	Laya	0	28.8–28.8	0	34.4–34.4
CLINC150	Laya	0	23.1–23.1	0	44.4–44.4
MASSIVE	Laya	0	23.8–23.8	0	33.0–33.0

Appendix

Table 13: Post-hoc sensitivity to exact top-probability ties. Counts use recorded normalized vectors. Accuracy ranges (percent) cover arbitrary choices among tied maxima, holding all non-tied predictions fixed; these are deterministic tie-resolution bounds.

Dataset	Model	10 bins	15 bins	20 bins
TREC	Jev	+0.019	+0.021	+0.020
CLINC150	Jev	+0.032	+0.028	+0.028
MASSIVE	Jev	+0.028	+0.026	+0.032
TREC	Laya	-0.038	-0.033	-0.038
CLINC150	Laya	-0.080	-0.080	-0.073
MASSIVE	Laya	+0.084	+0.078	+0.079

Appendix

Table 14: Post-hoc bin-count sensitivity: marginal-minus-flat top-label ECE with 10, 15, or 20 equal-width bins. Negative values indicate lower empirical ECE. These descriptive point estimates assess bin-count sensitivity.

Explore similar work

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration
Sep 28, 2026Riccardo PorceddaProbability Confidence Calibration
Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy
Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation
Sep 27, 2026Gowthamkumar NandakishoreEscalation Post-Hoc
Reported Confidence in LLMs Tracks Commitment More Than Correctness
Jun 28, 2026Dharshan KumaranLarge Language Model Reliability Correctness
Google DeepMind

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Riccardo Porcedda

Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy

Probability Confidence Calibration Confidence Estimation

Open paper AskPDF arXiv

The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition

A

for which the exact probability

P(A)

is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers,

P(A)

and

P(\neg A)

are presented as if

P(A)+P(\neg A)=1

, while a term

P(U)\neq0

is missing in the sum. Recovering

P(U)

leads to an improvement of median soft accuracy in \texttt{Choice} answers from

0.771

0.978

, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.

Abstract

Explore similar work

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Reported Confidence in LLMs Tracks Commitment More Than Correctness