Latent Coherence: Do Frontier Language Models' Probability Judgments Obey the Law of Total Probability?

Peter Potash

cs.CLOct 22, 2026

Latent Coherence: Do Frontier Language Models' Probability Judgments Obey the Law of Total Probability?

Authors: Peter Potash

Abstract

LawZero's Scientist AI trains a Predictor toward the Bayesian posterior, with an objective that penalizes inconsistent probabilities among latent explanations. We ask whether current frontier language models, with no such training, are already consistent in this sense zero-shot: do their probability judgments satisfy the law of total probability, $P(X)=\sum_z P(z)P(X|z)$ , when the latent $z$ must be inferred from records in the context? In separate single-turn calls, models are asked for $P(X)$ , for $P(z)$ over the latent's states, and for $P(X|z)$ ; a causal family applies the same test to $P(y|do(x))$ and the back-door adjustment. Every query has an exact ground-truth probability, and a subset of items is asked in three wordings to separate persistent incoherence from rewording noise. Across seven models, GPT-6.1 Sol and Claude Opus 5.5 are the most coherent (mean absolute residual 0.08 and 0.12 log-odds, about 2 percentage points). Claude Haiku 5.5, Gemini 3.8 Flash and DeepSeek V4-Pro show persistent incoherence (0.27-0.35 log-odds) that is mostly item-specific rather than a shared offset. Within every model, larger residuals go with larger errors against the truth (Spearman 0.40-0.62), so the residual, which needs no ground truth, flags unreliable answers. Reasoning length does not predict coherence, across models or within a model across rewordings of the same question. Simple consistency checks pass almost everywhere; the failures are specific to reasoning through a latent variable. Code and data: https://github.com/ppotash/latent-coherence

Figures & tables

Family	Records in context	Latent $Z$
source	Reports on a claim from 1–3 sources, each with a track record of checked past reports	Whether one source is careful or careless
diagnostic	A fault-diagnosis inspection log, with symptom rates for each of three causes	Which of three causes is present
causal	A count table over confounder $Z$ , treatment $X$ and outcome $Y$	The confounder (e.g. which production line a battery came from), adjusted for by the back-door formula ( 3 )
hover	A fictionalized HoVer claim split into sub-claims, with reports of the sub-claims from sources of varying reliability	Truth of the bridge sub-claim; a source’s kind

Table 1: Item families. source , diagnostic and hover come in a full version, which states every probability, and a qualitative version, which gives only verbal descriptions (“usually”, “most sources are careful”). Appendix A gives examples.

Model	Effort	Mean residual	Persistent	Noise	Spread	Dir.	Err. vs truth	Tok./ans.	Cost
GPT-6.1 Sol	med.	0.08 [0.06, 0.11]	0.08 ∗ [0.02, 0.13]	0.16	0.08	$-0.01$	0.34	400	$4
Claude Opus 5.5	med.	0.12 [0.07, 0.17]	0.00 [0.00, 0.14]	0.34	0.00	$+0.01$	0.38	500	$18
DeepSeek V4-Pro	high	0.15 [0.10, 0.21]	0.27 ∗ [0.10, 0.40]	0.30	0.26	$+0.08$	0.37	16,900	$53
Grok 4.7	low	0.21 [0.15, 0.27]	0.21 [0.00, 0.32]	0.51	0.20	$+0.05$	0.47	7,200	$70
Gemini 3.8 Flash	med.	0.26 [0.19, 0.34]	0.28 ∗ [0.10, 0.40]	0.50	0.26	$-0.11$	0.73	4,400	$15
GLM 5.3	high	0.26 [0.20, 0.33]	0.11 [0.00, 0.25]	0.53	0.09	$-0.07$	0.41	6,700	$54

Table 2: Main results on the latent total-probability identity, in log-odds, with 95% bootstrap CIs. Mean residual : average

|r|

over 90 items (first wording). Persistent , Noise , Spread and Dir. (signed mean residual): § 3.3 , on 83 noise-floor instances;

\text{Persistent}^{2}\approx\text{Spread}^{2}+(\text{Dir.})^{2}

. ∗ : the persistent residual’s CI excludes 0, i.e. persistent incoherence is detected. Err. vs truth : mean

|\operatorname{logit}\hat{P}(X)-\operatorname{logit}P(X)|

. Tok./ans. : mean reasoning tokens per answer. Cost : whole run (Haiku 1,890 prompts; DeepSeek at off-peak rates).

Figure 1: (a) Mean absolute residual over all 90 latent items. (b) Persistent residual on the noise-floor items. Bars are 95% bootstrap CIs; filled markers in (b) have CIs that exclude 0.

	Residual vs error against truth			Residual vs reasoning tokens
Model	Spearman $\rho$ [CI]	Err. if $\|r\|\leq 0.1$	Err. if $\|r\|>0.5$ ( $n$ )	Across items $\rho$ [CI]	Within item corr. [CI]
GPT-6.1 Sol	0.62 [0.45, 0.76]	0.25	0.56 (2)	$-0.22$ [ $-0.41$ , $-0.03$ ]	$-0.05$ [ $-0.19$ , 0.12]
Claude Opus 5.5	0.50 [0.33, 0.64]	0.24	0.65 (4)	$-0.04$ [ $-0.23$ , 0.13]	$\phantom{-}0.00$ [ $-0.14$ , 0.16]
DeepSeek V4-Pro	0.53 [0.36, 0.66]	0.29	0.75 (16)	$\phantom{-}0.17$ [0.00, 0.33]	$-0.08$ [ $-0.23$ , 0.08]
Grok 4.7	0.57 [0.40, 0.70]	0.24	1.15 (20)	$\phantom{-}0.03$ [ $-0.15$ , 0.20]	$\phantom{-}0.13$ [ $-0.07$ , 0.28]
Gemini 3.8 Flash	0.49 [0.30, 0.66]	0.41	1.68 (25)	$-0.10$ [ $-0.27$ , 0.07]	$\phantom{-}0.05$ [ $-0.15$ , 0.27]

Table 3: Within-model relations, over 130 latent-identity instances (first wording). Left : Spearman correlation of

|r|

with the direct answer’s error

|\operatorname{logit}\hat{P}(X)-\operatorname{logit}P(X)|

, and the mean error on coherent and incoherent instances. Right : Spearman correlation of

|r|

with reasoning tokens summed over the identity’s

1+2K

calls; and, on the 83 noise-floor instances, the Pearson correlation of

|r|

with log tokens after centering both within each instance, which compares the three wordings of the same question and so removes item difficulty. Its CI resamples whole instances, keeping each one’s three wordings together.

Figure 2: Mean absolute residual (95% CI) against mean reasoning tokens per answer (log scale).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	Effort (levels offered)	Mode	Prompts	Reasoning tok./ans.	Cost (USD)
Claude Haiku 5.5	Anthropic	medium	batch	1,890	1,117	0.62
Claude Opus 5.5	Anthropic	medium	batch	1,590	499	17.61
GPT-6.1 Sol	OpenAI	medium	batch	1,590	400	3.96
Gemini 3.8 Flash	Google	medium	batch	1,590	4,409	14.52
Grok 4.7	xAI	low (low/med./high/xhigh)	sync	1,590	7,236	70.44
GLM 5.3	Z.ai	high (low/high/max)	sync	1,590	6,733	53.57

Appendix

Table 4: Settings and cost. DeepSeek’s cost is at off-peak rates ($106.87 at peak rates). Batch runs are billed at half price.

Group ( $n$ identity instances)	Haiku	Opus	DeepSeek	Gemini	GLM	GPT	Grok
diagnostic / full (5)	0.002	0.006	0.000	0.000	0.002	0.000	0.002
source / full (5)	0.020	0.045	0.001	0.013	0.043	0.000	0.055
diagnostic / qualitative (20)	0.110	0.055	0.094	0.075	0.275	0.044	0.086
source / qualitative (20)	0.382	0.176	0.078	0.261	0.336	0.152	0.224
hover / full (40)	0.376	0.046	0.186	0.169	0.109	0.012	0.234
hover / qualitative (40)	0.798	0.230	0.307	0.641	0.438	0.164	0.368

Appendix

Table 5: Mean absolute residual (log-odds) by group, first wording.

	Full spec (32 instances)		Qualitative (51 instances)
Model	Noise	Persistent	Noise	Persistent
Claude Haiku 5.5	0.37	0.34 [0.00, 0.57]	0.77	0.35 [0.00, 0.52]
Claude Opus 5.5	0.07	0.03 [0.00, 0.06]	0.43	0.00 [0.00, 0.17]
DeepSeek V4-Pro	0.28	0.36 ∗ [0.04, 0.57]	0.31	0.19 [0.00, 0.31]
Gemini 3.8 Flash	0.40	0.09 [0.00, 0.20]	0.56	0.35 ∗ [0.11, 0.51]
GLM 5.3	0.45	0.00 [0.00, 0.00]	0.57	0.18 [0.00, 0.33]

Appendix

Table 6: Rewording noise and persistent residual by spec level (noise-floor items). ∗ : CI excludes 0.

Model	$n$	$\hat{P}(z)$	$\hat{P}(X\mid z)$	$\hat{P}(X)$	$D$	$n$	$\hat{P}(z)$	$\hat{P}(X\mid z)$	$\hat{P}(X)$	$D$	$D$ closer
	Coherent ( $\|r\|\leq 0.1$ )					Incoherent ( $\|r\|>0.5$ )
Claude Haiku 5.5	53	0.36	0.32	0.34	0.34	32	1.06	1.04	0.89	1.41	28%
Claude Opus 5.5	91	0.35	0.29	0.24	0.25	4	1.61	0.51	0.65	1.19	25%
DeepSeek V4-Pro	91	0.32	0.32	0.29	0.30	16	1.06	0.63	0.75	1.09	38%
Gemini 3.8 Flash	63	0.37	0.33	0.41	0.42	25	0.88	0.90	1.68	1.25	64%
GLM 5.3	59	0.25	0.21	0.21	0.23	26	0.76	0.55	0.85	0.86	50%

Appendix

Table 7: Error attribution: mean log-odds error against the truth of each estimate (averaged over latent states for

\hat{P}(z)

and

\hat{P}(X\mid z)

), for coherent and incoherent latent-identity instances (first wording).

D

: the model’s own decomposition.

D

closer : share of incoherent instances where

D

is closer to the truth than

\hat{P}(X)

. Pooled over models, errors on full-spec instances are 0.13 (

\hat{P}(z)

), 0.07 (

\hat{P}(X\mid z)

), 0.09 (

\hat{P}(X)

) and 0.11 (

D

); on qualitative instances, 0.69, 0.63, 0.69 and 0.72.

Model	Err. direct	Err. decomposition	Decomp. closer	Normalization (median $\|\Sigma-1\|$ , all checks)
Claude Haiku 5.5	0.50	0.64	44%	0.010
Claude Opus 5.5	0.38	0.42	43%	0.003
DeepSeek V4-Pro	0.37	0.44	40%	0.000
Gemini 3.8 Flash	0.73	0.66	49%	0.010
GLM 5.3	0.41	0.44	39%	0.020
GPT-6.1 Sol	0.34	0.33	47%	0.000

Appendix

Table 8: Accuracy against ground truth (mean log-odds error over 130 latent-identity instances) for the direct answer and for the model’s own decomposition, the share of instances where the decomposition is closer, and the median deviation from 1 pooled over all normalization checks.

A	B	$\Delta$	$p_{\text{Bonf}}$	A	B	$\Delta$	$p_{\text{Bonf}}$
Haiku	Opus	0.256	$\leq$ 0.001	DeepSeek	Gemini	$-$ 0.108	0.303
Haiku	DeepSeek	0.224	$\leq$ 0.001	DeepSeek	GLM	$-$ 0.112	0.168
Haiku	Gemini	0.116	0.541	DeepSeek	GPT	0.065	0.639
Haiku	GLM	0.112	0.819	DeepSeek	Grok	$-$ 0.058	1.000
Haiku	GPT	0.289	$\leq$ 0.001	Gemini	GLM	$-$ 0.004	1.000
Haiku	Grok	0.166	0.028	Gemini	GPT	0.173	$\leq$ 0.001

Appendix

Table 9: Paired item-level differences

\Delta

= mean

|r|_{A}

-

mean

|r|_{B}

over 90 latent items, with sign-flip permutation

p

-values (20,000 permutations) Bonferroni-corrected over 21 pairs. “

\leq

0.001” is the resolution floor of the test after correction.

	Answers (%)		Mean $\|r\|$ , clip at			$\|\hat{P}(X)-D\|$ (pp)		Latent states
Model	clipped	exactly 0/1	0.001	0.01	0.05	mean	median	renorm. $\|r\|$	med. $\|\sum_{k}\hat{P}(z_{k})-1\|$
GPT-6.1 Sol	7.4	5.7	0.084	0.083	0.083	1.5	0.3	0.071	0.000
Claude Opus 5.5	7.9	2.5	0.123	0.115	0.110	1.9	0.5	0.085	0.003
DeepSeek V4-Pro	8.2	5.8	0.162	0.148	0.125	2.1	0.3	0.143	0.000
Grok 4.7	7.7	6.1	0.209	0.206	0.192	3.0	1.4	0.188	0.001
Gemini 3.8 Flash	9.6	7.1	0.371	0.255	0.206	3.5	1.6	0.251	0.012

Appendix

Table 10: Robustness. Clipped : share of all 1,590 answers at or beyond 0.01 or 0.99. Mean

|r|

: the headline measure (item-level mean, latent identities) with clip bounds

[\epsilon,\ 1-\epsilon]

|\hat{P}(X)-D|

: the residual on the probability scale, in percentage points. Latent states : mean

|r|

after rescaling the

\hat{P}(z_{k})

to sum to 1 (item-level mean over 90 items, like the headline measure), and the median deviation of their sum from 1 over the 130 latent-identity instances (first wording). The latter’s base differs from the normalization column of Table 8 , which pools all of the harness’s normalization checks, so the medians differ slightly.

Reasoning Consistency in Language Models

Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.

See all 123 papers

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?
Sep 27, 2026Keyi Li, Yihao He, Quanyi LiReasoning Consistency in Language Models
Software College, Northeastern University Shenyang, China
Probabilistic Attribution For Large Language Models
May 20, 2026Shilpika Shilpika, Carlo Graziani, Bethany Lusch +2LLM Interpretability LLM Uncertainty Estimation
Argonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA · Mathematics and Computer Science Division Argonne National Laboratory Lemont, IL, USA · Department of Computer Science University of Illinois Chicago Chicago, IL, USA
Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models
Sep 22, 2026Xiaoyu Luo, Tao Ren, Wenrui Yu +3LLM Interpretability CoT Reasoning
Department of Computer Science Aalborg University, Copenhagen, Denmark · Department of Electronic Systems Aalborg University, Copenhagen, Denmark · Seafill

Sep 27, 2026J/K move · Enter open · S save

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Keyi Li, Yihao He, Quanyi Li

Software College, Northeastern University Shenyang, China

Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items.

Abstract

Reasoning Consistency in Language Models

Explore similar work

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Probabilistic Attribution For Large Language Models

Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models