Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0), potentially exceeding nominal ε; (iii) the within-unit mean's effective sample size saturates at 1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ^=0.435, θ^=+0.96) the gap is dormant, whereas on AIME (ρ^=0.480, θ^=−0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.
Figures & tables
Result
Target
Filtration
Dependence
Type
Thm 4.2 (realized validity)
H0rel:μ(Z)≤0
oracle Gn (cond. on Z )
exchangeable
finite-sample, anytime
Prop 4.5 (over-certification)
H0pop:θ≤0
observed Fn
exch., π>0
asymptotic lower bound
Thm 5.1 (saturation)
estimation of θ
observed Fn
exch., ρ>0
exact variance, a.s. limit
Thm 6.2 (remedy)
H0pop:θ≤0
unit-level Hq
indep. across units
finite-sample, anytime
Table 1: Scope of the main results. “Anytime” denotes exact, finite-sample validity at data-dependent stopping times. None of the validity guarantees requires knowing ρ ; ρ governs only efficiency (betting fraction and allocation).
Quantity
Single-unit certificate
Prompt-level remedy
Recovered ρ (ICC)
0.091 (planted 0.091 )
Var(Yˉn) limit
0.090=σB2 (saturates)
neff limit
≈11≈1/ρ
Realized-null Type I
0.005
N/A
Population-null certification rate
0.32(→π=0.37)
0.033 at θ=0
Table 2: Synthetic study (population null true, θ≤0 , π=0.37 , planted ρ=0.09 ). The single-unit certificate controls the realized null but over-certifies the population null toward π ; the prompt-level remedy restores population validity, including at the least-favorable θ=0 .
Benchmark (prompts × rollouts)
ρ^ (95% CI)
1/ρ^
θ^
π^
median agr.
GSM8K ( 300×30 )
0.435[0.208,0.619]
2.3
+0.96
0.99
1.00
AIME ( 150×30 )
0.480[0.405,0.544]
2.1
−0.15
0.33
0.30
Table 3: Across-prompt correlation and the boundary, two benchmarks (Claude Haiku 4.5, temperature 1 , split-sample leader). On both, ρ^ is bounded well away from zero. The benchmarks differ in θ : GSM8K is easy and the leader is strongly favored on average, so the realized–population gap is structurally present but operationally invisible; AIME is hard and the leader is not favored on average, placing the negative estimated population margin and positive estimated π , the regime where over-certification can occur.
Quantity (whole benchmark; θ^=−0.15 , π^=0.33 )
Rate
Single-unit certification rate (per prompt)
0.240
Prompt-level population certification rate (bootstrap)
0.000
Nominal level ϵ
0.05
Table 4: Over-certification on AIME (Claude Haiku 4.5, 150 prompts). At the whole-benchmark level the estimated population margin is negative ( θ^=−0.15 ), yet single-unit certificates fire on nearly one prompt in four, while the prompt-level remedy declines.
Pilot-agreement bin
n
θ^
π^
ρ^
single-unit rate
[0.0,0.3]
60
−0.70
0.03
0.11
0.00
(0.3,0.5]
37
−0.34
0.16
0.14
0.03
(0.5,0.7]
16
−0.14
0.25
0.11
0.06
(0.7,0.9]
11
+0.74
1.00
0.07
0.73
(0.9,1.0]
26
+0.97
1.00
0.04
1.00
Table 5: AIME by difficulty (pilot-agreement bins; estimates on held-out). As difficulty falls (left to right), θ sweeps from −0.70 to +0.97 while π and the single-unit certification rate track the realized side. The drift crosses zero between the (0.5,0.7] and (0.7,0.9] bins, where the single-unit rate jumps from 0.06 to 0.73 .
Figure 1: The empirical wedge on AIME. Across pilot-agreement bins (hard at left, easy at right): mean held agreement (θ+1)/2 and π=P(μq>0) rise together, while the single-unit certification rate stays near zero across the low-agreement bins, which hold most of the mass and pull the population drift to θ^=−0.15 , and climbs to 1.00 only on the high-agreement minority. That minority is the 24% the single-unit certificate fires on, even though the estimated population margin is negative.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.