stat.MENov 19, 2026

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

Authors: Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe

Organizations: Department of Economics, Hunter College and Graduate Center, CUNY · Department of Statistics, Rutgers University

Abstract

Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0)π=P(μ(Z)>0), potentially exceeding nominal εε; (iii) the within-unit mean's effective sample size saturates at 1/ρ1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ^=0.435\hatρ=0.435, θ^=+0.96\hatθ=+0.96) the gap is dormant, whereas on AIME (ρ^=0.480\hatρ=0.480, θ^=−0.15\hatθ=-0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.

Figures & tables

Explore similar work

CardsList
  1. CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency

    May 7, 2026Hirofumi Ota, Naoto Iwase, Yuki Ichihara +2LLM EvaluationAnytime-Valid Inference

  2. How Uncertainty Estimation Scales with Sampling in Reasoning Models

    Mar 19, 2026Maksym Del, Markus Kängsepp, Marharyta Domnich +4LLM Uncertainty EstimationUncertainty Quantification

  3. Reliable Chain-of-Thought via Prefix Consistency

    May 8, 2026Naoto Iwase, Yuki Ichihara, Mohammad Atif Quamar +1Self-Consistency DecodingReasoning Consistency in Language Models