Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Keyi Li; Yihao He; Quanyi Li

cs.CLSep 27, 2026

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Authors: Keyi Li, Yihao He, Quanyi Li

Organizations: Software College, Northeastern University Shenyang, China

Abstract

Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items.

Figures & tables

Figure 1: Negation pairs. Each point is one pair

(s_{k},n_{k})

for an item and label, and point area is proportional to the number of pairs at that grid value. Coherent answers lie on the line

s_{k}+n_{k}=1

, and the band marks deviations of at most 0.1. Pairs in the shaded lower-left quadrant answer “no” to both “the label is

X_{k}

” and “the label is not

X_{k}

”, and pairs in the upper-right quadrant answer “yes” to both. MAD is the item-averaged

|s_{k}+n_{k}-1|

with its 95% bootstrap CI. All three panels show the same 480 pairs.

	MAD, mean $\|s_{k}+n_{k}-1\|$			Signed deviation	Same-side
System	All (160 items)	NLI (100)	PubMedQA (60)	mean $(s_{k}+n_{k}-1)$	rate
Jev (native)	$0.064$ $[0.055,0.072]$	$0.080$ $[0.068,0.092]$	$0.036$ $[0.026,0.047]$	$+0.054$ $[0.045,0.063]$	$0.067$ $[0.046,0.090]$
Qwen3.8-27B (first-token)	$0.293$ $[0.270,0.316]$	$0.306$ $[0.279,0.334]$	$0.270$ $[0.230,0.310]$	$-0.264$ $[-0.291,-0.238]$	$0.417$ $[0.367,0.465]$
Difference (Jev $-$ Qwen)	$-0.229$ $[-0.253,-0.206]$	$-0.226$ $[-0.257,-0.195]$	$-0.234$ $[-0.272,-0.196]$	$+0.318$ $[0.290,0.346]$	$-0.350$ $[-0.402,-0.298]$
Qwen3.8-27B (verbalized)	$0.122$ $[0.098,0.148]$	–	–	$+0.054$ $[0.029,0.081]$	$0.119$ $[0.092,0.148]$

Table 1: Negation complementarity. Item means with 95% bootstrap CIs (160 items, 480 pairs). The same-side rate is the share of pairs answered on the same side of 0.5.

Main comparison and confound checks
Comparison	Items (pairs)	Jev	Qwen	Difference [95% CI]
Main comparison	160 (480)	$0.064$	$0.293$	$-0.229$ $[-0.253,-0.206]$
Jev repeats averaged	160 (480)	$0.063$	$0.293$	$-0.229$ $[-0.253,-0.206]$
Confidence-matched pairs	140 (253)	$0.028$	$0.257$	$-0.229$ $[-0.264,-0.195]$
Qwen verbalized readout	160 (480)	$0.064$	$0.122$	$-0.058$ $[-0.083,-0.035]$
Double-negation labels removed	160 (320)	$0.057$	$0.385$	$-0.328$ $[-0.360,-0.297]$

Table 2: Robustness of the difference in negation violation,

\mathrm{MAD}_{\text{neg}}

(Jev)

-

\mathrm{MAD}_{\text{neg}}

(Qwen3.8-27B), for Qwen3.8-27B’s first-token readout unless stated. Confidence-matched pairs satisfy

|s^{J}_{k}-s^{Q}_{k}|\leq 0.10

; choice-matched pairs

|c^{J}_{k}-c^{Q}_{k}|\leq 0.10

. Matched subsets are averaged within items first. The double-negation labels are contradiction and negative . “Mass” is the probability the first-token readout assigns to the answer labels before renormalization, and a pair is dropped when either of its readouts falls below the bound. The logit row reports

|\mathrm{logit}\,s_{k}+\mathrm{logit}\,n_{k}|

. The first block gives the main comparison and the checks for the four confounds of Section 3 .

Identity	Statistic	Jev	Qwen (first-token)	Difference
Negation	$\|s_{k}+n_{k}-1\|$	$0.064$ $[0.055,0.072]$	$0.293$ $[0.270,0.316]$	$-0.229$ $[-0.253,-0.206]$
	$s_{k}+n_{k}-1$	$+0.054$ $[0.045,0.063]$	$-0.264$ $[-0.291,-0.238]$	$+0.318$ $[0.290,0.346]$
Disjunctive complement	$\|s_{k}+d_{k}-1\|$	$0.054$ $[0.048,0.061]$	$0.343$ $[0.319,0.369]$	$-0.289$ $[-0.315,-0.264]$
	$s_{k}+d_{k}-1$	$+0.004$ $[-0.004,0.012]$	$-0.327$ $[-0.356,-0.299]$	$+0.331$ $[0.302,0.360]$
Partition additivity	$\|\sum_{k}s_{k}-1\|$	$0.141$ $[0.123,0.159]$	$0.319$ $[0.288,0.352]$	$-0.178$ $[-0.214,-0.144]$
	$\sum_{k}s_{k}-1$	$+0.141$ $[0.123,0.159]$	$-0.286$ $[-0.324,-0.250]$	$+0.427$ $[0.384,0.470]$

Table 3: Other coherence identities (160 items). Item means with 95% bootstrap CIs. The difference is Jev minus Qwen3.8-27B. For monotonicity,

q

is the consequent probability (

d_{k}

n_{k}

m\neq k

), and the share counts checks violated by more than 0.05.

System	Question form	Target	All	NLI	PubMedQA
Jev (native)	$S_{k}$ (is $X_{k}$ )	$1/3$	$+0.047$ $[0.041,0.053]$	$+0.045$ $[0.038,0.052]$	$+0.050$ $[0.040,0.062]$
	$N_{k}$ (is not $X_{k}$ )	$2/3$	$+0.007$ $[0.002,0.013]$	$+0.029$ $[0.021,0.037]$	$-0.029$ $[-0.036,-0.023]$
	$D_{k}$ (is $X_{i}$ or $X_{j}$ )	$2/3$	$-0.043$ $[-0.048,-0.039]$	$-0.043$ $[-0.049,-0.038]$	$-0.043$ $[-0.050,-0.037]$
Qwen (first-token)	$S_{k}$ (is $X_{k}$ )	$1/3$	$-0.095$ $[-0.108,-0.083]$	$-0.117$ $[-0.132,-0.102]$	$-0.059$ $[-0.081,-0.037]$
	$N_{k}$ (is not $X_{k}$ )	$2/3$	$-0.169$ $[-0.185,-0.153]$	$-0.173$ $[-0.190,-0.156]$	$-0.162$ $[-0.194,-0.130]$
	$D_{k}$ (is $X_{i}$ or $X_{j}$ )	$2/3$	$-0.232$ $[-0.251,-0.213]$	$-0.271$ $[-0.294,-0.249]$	$-0.165$ $[-0.200,-0.131]$

Table 4: Decomposition by question form (exploratory). Mean probability of “yes” per question form minus the value that any coherent assignment implies when exactly one of three labels applies (Target), with 95% bootstrap CIs. The verbalized readout was asked only the

S_{k}

and

N_{k}

questions.

Figure 2: Negation violations by label, with 95% bootstrap CIs (100 NLI and 60 PubMedQA items). † Labels that denote falsity, whose negation is a double negative. (a) Mean

|s_{k}+n_{k}-1|

. (b) Mean

s_{k}+n_{k}-1

, where negative values mean that the two answers of a pair together give too little probability to “yes”.

Figure 3: (a, b) Mean negation violation by the confidence of the answer to

S_{k}

|s_{k}-0.5|

, on the probability and logit scales, with the number of pairs above the bars in (a) and 95% intervals from a bootstrap over items. (c) Item-level violation against the entropy of 100 human labels on ChaosNLI. Lines are running means over 25 items, and

\rho

is the Spearman correlation with its 95% bootstrap CI.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

		$\|s_{k}+n_{k}-1\|$			$s_{k}+n_{k}-1$		$d_{k}-n_{k}$		logit MAD
Domain	Label $X_{k}$	Jev	Qwen	Difference [95% CI]	Jev	Qwen	Jev	Qwen	Jev	Qwen
NLI	entailment	$0.078$	$0.477$	$-0.398$ $[-0.444,-0.351]$	$+0.075$	$-0.472$	$-0.057$	$-0.043$	$0.57$	$3.77$
	neutral	$0.063$	$0.336$	$-0.274$ $[-0.317,-0.231]$	$+0.052$	$-0.324$	$-0.022$	$-0.159$	$0.36$	$1.85$
	contradiction †	$0.100$	$0.105$	$-0.005$ $[-0.037,0.025]$	$+0.095$	$-0.071$	$-0.138$	$-0.094$	$0.63$	$1.26$
PubMedQA	affirmative	$0.033$	$0.372$	$-0.339$ $[-0.392,-0.285]$	$+0.012$	$-0.305$	$-0.022$	$+0.080$	$0.23$	$2.94$
	negative †	$0.041$	$0.114$	$-0.073$ $[-0.107,-0.039]$	$+0.027$	$-0.038$	$-0.014$	$-0.099$	$0.28$	$0.80$

Appendix

Table 5: Negation violations by (domain, label). † Double-negation labels. The logit MAD is the mean of

|\mathrm{logit}\,s_{k}+\mathrm{logit}\,n_{k}|

with probabilities clipped to

[0.01,0.99]

Quantity	Value
Jev calls (single-question + bundled)	3,360 (3,360 succeeded)
Qwen first-token calls	1,720 (1,720 parsed)
Qwen verbalized calls	960 (960 parsed)
Jev total cost	$0.0757
Jev latency, median / p95 (s)	0.64 / 0.75
Qwen first-token latency, median / p95 (s)	0.44 / 0.90

Appendix

Table 6: Calls, readout quality, repeat noise, and cost. During data collection the Qwen3.8-27B model server handled no other requests and its GPUs ran no other jobs. Its latency is the wall-clock time of a first-token call with four calls in flight.

Reasoning Consistency in Language Models

Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.

See all 121 papers

Labels Override Definitions in Jev-Style Typed Decision Models
Oct 1, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLanguage Model Bias Evaluation Prompt Sensitivity
University of Southern California (USC), USA
Do System One Decisions Add Up? A Study of Probabilistic Coherence
Sep 27, 2026Saman Sarker JoyLLM Evaluation Classification
Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia
Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration
Sep 28, 2026Riccardo PorceddaProbability Calibration Uncertainty Calibration
Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy

Oct 1, 2026J/K move · Enter open · S save

Labels Override Definitions in Jev-Style Typed Decision Models

Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram

University of Southern California (USC), USA

Language Model Bias Evaluation Prompt Sensitivity

Open paper AskPDF arXiv

A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply. Jev introduced this interface for routing, moderation and triage, open implementations followed, and the same operation occurs whenever a language model is used as a classifier by scoring label strings. We study the open implementations, whose weights we can inspect and patch, and ask whether the probability follows the definitions or the labels. A preference for the label we call option-label bias. Across four open-weight typed decision models, three ways of reading an answer from a Qwen2.5 backbone, eleven classification tasks and PolicyBench, a synthetic routing suite we introduce in which the rule appears only in the definitions, the answer is mostly the labels. Deleting every definition leaves accuracy unchanged (laya-td: 0.8559 against 0.8487), although those definitions support 0.7971 on their own, and renaming the options to A and B raises accuracy by +0.1511 [+0.1377, +0.1646]. One system, von, is unaffected, and the two code bases differ in one expression: laya writes each option as "{label}: {definition}", while von writes only the definition. Changing that expression in both directions, with no weight changed, makes all three laya checkpoints exactly invariant (+0.0000 [+0.0000, +0.0000]) and creates the effect in von, whose accuracy falls from 0.8511 to 0.2281 when a label contradicts its definition. Earlier work attributed this failure to the constrained decision head these models use in place of a text decoder; our results locate it in the prompt rendering. We give a two-call test that tells a practitioner which case applies to their model, and measure what four mitigations are worth.

Abstract

Reasoning Consistency in Language Models

Explore similar work

Labels Override Definitions in Jev-Style Typed Decision Models

Do System One Decisions Add Up? A Study of Probabilistic Coherence

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration