stat.MENov 19, 2026

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

Authors: Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe

Organizations: Department of Economics, Hunter College and Graduate Center, CUNY · Department of Statistics, Rutgers University

Abstract

Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0)π=P(μ(Z)>0), potentially exceeding nominal εε; (iii) the within-unit mean's effective sample size saturates at 1/ρ1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ^=0.435\hatρ=0.435, θ^=+0.96\hatθ=+0.96) the gap is dormant, whereas on AIME (ρ^=0.480\hatρ=0.480, θ^=−0.15\hatθ=-0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.

Figures & tables

Explore similar work

May 7, 2026

CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency

Large language models often improve reasoning by sampling multiple outputs and aggregating their final answers, but precise and efficient control of error levels remains a challenging task. In particular, deciding when to stop sampling remains difficult when the stopping rule is data-dependent and the set of possible answers is not known in advance. We study anytime-valid certification of a prespecified target answer as the unique mode of the model's response distribution, a guarantee distinct from answer correctness. We propose the Certification by Intersection-union Testing with E-processes (CITE) algorithm, which provably controls false certification at any prescribed level under arbitrary data-driven stopping, without requiring prior knowledge of the answer category set. We also prove an category-set-size-free stopping-time rate, establish matching minimax lower bounds up to constants in the main regime, and extend the construction to confidence-weighted voting. Simulations and LLM self-consistency experiments show empirical error control and improved certification in diffuse-tail settings.
Mar 19, 2026

How Uncertainty Estimation Scales with Sampling in Reasoning Models

Uncertainty estimation is critical for deploying reasoning language models, yet remains poorly understood under extended chain-of-thought reasoning. We study parallel sampling as a fully black-box approach using verbalized confidence and self-consistency. Across three reasoning models and 17 tasks spanning mathematics, STEM, and humanities, we characterize how these signals scale. Both self-consistency and verbalized confidence scale in reasoning models, but self-consistency exhibits lower initial discrimination and lags behind verbalized confidence under moderate sampling. Most uncertainty gains, however, arise from signal combination: with just two samples, a hybrid estimator improves AUROC by up to +12+12 on average and already outperforms either signal alone even when scaled to much larger budgets, after which returns diminish. These effects are domain-dependent: in mathematics, the native domain of RLVR-style post-training, reasoning models achieve higher uncertainty quality and exhibit both stronger complementarity and faster scaling than in STEM or humanities.
May 8, 2026

Reliable Chain-of-Thought via Prefix Consistency

Large language models often improve accuracy on reasoning tasks by sampling multiple chain-of-thought (CoT) traces and aggregating them with majority voting, a test-time technique called self-consistency. When we truncate a CoT partway through and regenerate the remainder, we observe that traces with correct answers reproduce their original answer more often than traces with wrong answers. We use this difference as a reliability signal, prefix consistency, that weights each candidate answer by how often it reappears under regeneration. It requires no access to token log-probabilities or self-rating prompts. Across five reasoning models and four math and science benchmarks, prefix consistency is the best correctness predictor in most settings, and reweighting votes by this signal reaches the plateau accuracy of majority voting with up to 21×21\times fewer tokens (median 4.6×4.6\times). Our code is available at https://github.com/naoto-iwase/prefix-consistency.