Statistical Hypothesis Testing

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 71

Oct 7, 2026cs.LG

What Can a Gaussian Process Design Test

A Gaussian process (GP) model can agree with the data for two reasons: its assumptions are right, or the chosen inputs could never have shown that they are wrong. The distinction can be checked from the design before any responses are observed. Every model implies relations that its noiseless responses must satisfy at the chosen inputs, such as the middle value lies on the line through its two neighbours. For GPs built from finitely many features, these relations are exactly the null space of the kernel matrix. Gale duality gives them a geometric interpretation, in which each observation has a vector and the smallest groups of observations that can expose an error are the circuits. For other kernels the relations become soft: response patterns may be improbable under the prior rather than algebraically impossible. A standard test then combines two kinds of evidence. Structural evidence comes from a violated relation and grows without limit as the noise falls. Prior-based evidence only says that a departure is improbable under the prior. With all inputs at the two ends of an interval, for example, a GP can reject a straight line against a large curvature, but only because the implied intercept is improbable, never because curvature was seen. In simulations the predicted power matched the observed rejection rates. Choosing the next input by predicted power raised the power against a localised discrepancy from 0.48 to 0.72, against 0.51 when choosing by predictive variance, and a grid in two dimensions contained exact tests of additivity that a Latin hypercube lacked. The test itself is classical. The contribution is the prospective reading of that test: before observing the responses, the design already determines what kind of contradiction it can produce.

Ivan De Boi, Marnix Van Soom

Department of Electromechanical Engineering University of Antwerp Campus Groenenborger – Z373 Groenenborgerlaan 171, 2020 Antwerp, Belgium · Artificial Intelligence Lab Vrije Universiteit Brussel Pleinlaan 2, 1050 Brussels, Belgium

Gaussian Processes Statistical Hypothesis Testing Bayesian Experimental Design

Oct 6, 2026stat.ML

Two-Sample Testing via Generative Processes

Deciding whether two samples come from the same distribution is a classical problem in statistics, and generative transport offers a new way to approach it. We build a stochastic interpolant directly between the two samples and observe that, for a symmetric schedule, its law is invariant under the time reflection

t \mapsto 1-t

whenever the two distributions coincide. We therefore test whether the marginals at times t and 1-t agree by computing their Jensen--Shannon divergence. Both marginals are explicit mixtures over all cross-pairs of observations, so nothing is learned, and permutation calibration gives an exact finite-sample level. For Gaussian noise, this divergence equals a time integral that pairs the reflection defects of the velocity field and of the score, so the test compares transport dynamics rather than endpoints alone. With a narrow-plus-broad noise design, the test attains the minimax separation rate n^{-2s/(4s+d)} over bounded, compactly supported densities whose difference has Sobolev smoothness s > 3d/4, with no lower bound on the densities. Fusing a dyadic grid of noise scales through their permutation ranks, without sample splitting, preserves exact level and adapts to unknown s at an iterated-logarithmic cost. Empirically, the test matches or outperforms state-of-the-art kernel two-sample tests.

Eshant English, Kenji Fukumizu, Taiji Suzuki

The University of Tokyo, Tokyo, Japan · Hasso Plattner Institute for Digital Engineering, Germany · Institute of Statistical Mathematics, Japan +1

Two-Sample Testing Statistical Hypothesis Testing

Oct 5, 2026quant-ph

Finding Gaussian Structure in Bosonic States

We study agnostic tomography of pure bosonic Gaussian states: given copies of an arbitrary

n

-mode bosonic state

ρ

, the goal is to output a pure Gaussian state whose infidelity with

ρ

is at most

\mathrm{opt} + ε

, where

\mathrm{opt}

is the minimum infidelity achievable by any pure Gaussian state. We give efficient protocols achieving this in both the high and low fidelity regimes. When

\mathrm{opt}

is below some universal constant, our protocol has runtime and copy complexity which is strongly polynomial in

n, 1/ε

and

\log \log E

, where

E

is the energy of the closest pure Gaussian state. For arbitrary

\mathrm{opt}

, our protocol uses

(n+1)^{\mathrm{poly}(1/ε)} \mathrm{poly}\left(1+\log\log(E)\right)

copies and runtime. As a corollary, we obtain the first truly tolerant Gaussianity testing protocol for distinguishing whether

\mathrm{opt} > c + ε

or

\mathrm{opt} < c - ε

, for any threshold

c\in(0,1)

. We also prove

\mathrm{poly}(n,1/ε)

runtime is impossible, unless

\mathrm{NP}\subseteq\mathrm{BQP}

. Our protocols follow a shared paradigm: first, we iteratively use general Gaussian measurements combined with techniques from classical robust statistics to obtain a good warm start estimate, then we leverage non-Gaussian measurements to refine this warm start using convex and non-convex optimization methods. Interestingly, we prove that non-Gaussian measurements are necessary to match the strong agnostic guarantees we obtain, and in fact these guarantees are provably superior to what is possible for robustly estimating classical Gaussians.

Alvan Arulandu, Sitan Chen, Ziyun Chen +2

Harvard University · University of Washington · Stanford University

Quantum State Tomography Statistical Hypothesis Testing

Oct 5, 2026stat.ML

Two-Sample Testing via Path-based Inference

Modern deep generative models are primarily studied for their ability to generate realistic samples, yet the generative dynamics they learn can also serve as objects of statistical inference. We develop this idea for two-sample testing, the problem of deciding whether the same distribution generated two finite datasets. Using stochastic interpolants, we connect both distributions to a shared Gaussian bottleneck, so that each half of the resulting path is a Gaussian channel acting on a single population. We prove that the null hypothesis holds if and only if the population denoiser, or equivalently, the velocity fields of the two halves, coincide at any single noise level, which amounts to a reflection symmetry of the path about the bottleneck. Deviations from this symmetry yield a continuum of two-sample witnesses, which we estimate via held-out regression risks on learned denoisers and velocities and aggregate along the path; under an information-theoretic weighting, the aggregated discrepancy equals the Jeffreys divergence between the noise-smoothed distributions. Calibrating the resulting statistics by permutation yields tests that are valid in finite samples for any trained networks and consistent when the fields are learned accurately. On a synthetic benchmark and three image benchmarks, the proposed tests improve power over the strongest baseline by up to 33 percentage points at an equal total sample budget, with the best choice of regression representation and path weighting depending on the data modality. These results show that generative paths provide a principled representation for statistical testing, extending stochastic-interpolant models beyond generation.

Eshant English, Wei-Cheng Lai, Yanfeng Yang +3

The University of Tokyo, Tokyo, Japan · Hasso Plattner Institute for Digital Engineering, Germany · Institute of Statistical Mathematics, Japan +3

Two-Sample Testing Statistical Hypothesis Testing

Sep 30, 2026cs.AI

Routing Probes Can Improve Without New Information: An Exact-Null Audit of Uncertainty Beyond Model Outputs

Routing signals of modern vision transformers -- expert gates, attention-residual weights and halting scores -- often improve probes that predict whether the model is correct, and the improvement is commonly read as evidence that routing carries information about errors beyond the model's outputs. We test this inference directly: keeping real output-routing pairs, we redraw correctness labels from a frozen output-only generator fitted on disjoint data, so that routing is uninformative by construction. Under this exact label null, a width-matched MLP comparison still reports a routing gain in 51.3% of confidence-only evaluations (308/600), while a linear comparison reports none. Holding each training trajectory fixed on a six-model panel and selecting the checkpoint by validation log loss instead of validation accuracy removes the detections (50/120 to 0/120, and 83/120 to 0/120 in an independently implemented probe), identifying accuracy-based checkpoint selection as the cause; across all output views the raw detection rate falls from 27.5% (528/1,920) to zero observed detections. The repaired comparison is not sensitive, detecting an implanted signal of about 0.005 nats in 0/20 replicates in each of two matched settings, whereas a conditional permutation test built on an estimated routing law detects it in 11/20 and 10/20 and rejects rarely under the null. On real correctness labels, the conditional analysis yields model-relative evidence in five DeiT attention-residual families; in four it persists under two specified variants of the conditional law, and no family passes an additional noise criterion. Fitting a better probe and testing for incremental information are different problems, and each needs its own validation.

Wenhao Liang, Lin Yue, Wei Emma Zhang +3

Vision Transformer Statistical Hypothesis Testing

Sep 24, 2026stat.ML

Robust Detection of LLM-Generated Text under Contamination

We study the detection of LLM-generated text under editing and contamination. Modeling human and machine text as finite-order Markov processes with Huber contamination, we characterize an exact boundary for reliable detection under our assumptions. Detection is impossible when contamination is sufficiently large relative to clean-source separation. Below this boundary, a collection of clipped likelihood-ratio tests achieves vanishing worst-case errors. This construction motivates clipping as a simple modification of existing statistical detectors. For a broad class of additive scores, we identify conditions under which the clipped test is consistent while the raw test's worst-case power tends to zero. We evaluate seven detectors across three datasets and three generation models, and on the RAID benchmark. Clipping improves robustness in both studies, with gains varying across detectors and contamination settings. For example, at a target false-positive rate of 5%, clipping improves the log-likelihood--log-rank ratio (LRR) detector's true-positive rate by a median of 8.3 percentage points in the controlled study and 2.1 and 4.3 points in rate- and attack-specific RAID evaluations, respectively.

Jiaxun Li, Saptarshi Chakraborty, Ambuj Tewari

Department of Statistics, University of Michigan

AI-Generated Text Detection Language Model Robustness Robustness of Machine-Generated Text Detection Statistical Hypothesis Testing

Sep 24, 2026cs.AI

Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation

Large language models are increasingly used as inexpensive judges to evaluate outputs, label data, and assess whether a system meets a desired quality standard. Yet using AI judgments for formal statistical inference is fundamentally different from simply treating them as ground-truth labels: AI evaluations can be biased or noisy, and rigorous hypothesis testing requires explicit control of type-I and type-II errors. We study how to use AI judgments, together with selective human verification, to conduct a valid hypothesis test at minimum cost. We consider a population of items with hidden binary labels. After choosing a fixed pool of items, the decision maker can selectively query AI, send an item directly to a human, escalate an AI-scored item to a human after observing the AI report, or stop once sufficient evidence has accumulated. We derive an information-theoretic lower bound that captures the minimum cost of achieving prescribed testing errors and characterizes the value of AI information and human verification through a report-dependent information frontier. Motivated by this characterization, we develop SCALE, a sequential cost-aware policy that combines selective AI scoring with adaptive human escalation. SCALE is valid at finite sample sizes and matches the lower bound to first order as the target error probabilities vanish. We further extend the framework to an unknown AI-output model using paired AI-human pilot data. Numerically, SCALE approaches Human-only or AI-only testing when one source clearly dominates, while achieving its largest savings when inexpensive AI judgments and selective human verification are both valuable.

Dae Woong, Ham, Xuejun Zhao +2

David · University of Michigan, Ann Arbor, MI 48109, USA · University of North Carolina at Charlotte, Charlotte, NC 28223, USA

Cost-Aware Inference Human-in-the-Loop Evaluation Sequential Hypothesis Testing Statistical Hypothesis Testing

Sep 23, 2026stat.ML

Selective Inference for Deep Clustering in Latent Spaces

Deep clustering is a powerful approach for discovering meaningful structures in high-dimensional data by learning a low-dimensional latent representation prior to clustering. Despite its empirical success, assessing the statistical reliability of the resulting clusters remains challenging. Testing discovered clusters on the same data induces selection bias and invalidates classical

p

-values. Selective inference (SI) provides a principled framework for correcting this bias, but existing methods focus on clustering performed directly on the observed features. In this work, we develop an SI framework for deep clustering with a fixed pretrained encoder. The key challenge is that cluster assignments are determined through a nonlinear transformation from the original data space to the latent space, resulting in a substantially more complex selection process than in conventional clustering. Our method provides a computationally tractable way to account for this process and enables valid statistical testing of differences between clusters identified in the latent space. Synthetic experiments demonstrate that the proposed method controls the Type I error rate while achieving higher power than valid but conservative baselines, and genomic applications show that it can identify significant cluster differences while appropriately accounting for selection bias. Our framework provides a principled approach to quantifying the statistical reliability of structures discovered by deep clustering.

Eina Mizui, Tomohiro Shiraishi, Shunichi Nishino +1

Nagoya University · RIKEN

Selective Inference Clustering Unsupervised Clustering Deep Clustering Statistical Hypothesis Testing

Sep 12, 2026cs.AI

When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents

Independent evaluation can reject harmful policy updates yet also prevent useful continual learning. We argue that update admission must be assessed through both error control and retained learning opportunities at a stated interaction budget. We identify a concrete failure: a range-based confidence gate cannot certify unchanged old-task behavior within otherwise substantial budgets. A standard paired-binomial construction reduces this burden when outcome disagreements are rare. We also specify certified historical-reference promotion and a round-level missed-opportunity metric. In a constructed one-step pushing diagnostic with 32 seeds, fresh paired checks admit 31.6% of a common update stream at 2,000 episodes per stage, versus zero for the range-based gate; unconditional replay nevertheless learns better in closed-loop runs. A separate learned-dynamics stress test distinguishes model bias from feedback-selection error. The contribution is an admission-audit protocol with analytical and synthetic evidence; physical-robot and VLA validation remain open.

Qinzhen Ma, Ruihai Wu

Rice University · University of California, Berkeley

Continual Robot Learning Safe RL Continual RL Statistical Hypothesis Testing

Sep 9, 2026stat.ME

Likelihood-free inference with nuisance parameters through normalizing flows

We present a simple decomposition of a neural-network-based normalizing flow that naturally uncovers a pivotal statistic (or something close) in the presence of nuisance parameters, based only on a sample generator from the distribution of interest. We show that the statistic is near-pivotal in the sense of minimum average KL-divergence of its

p

-values versus uniform and we argue that it can be expected to have good power when the dimension of the statistic equals the dimension of the parameter. It is able to incorporate prior knowledge about group invariances such as translation and scale. It can discover the one-sample

t

-test almost exactly, outperforms the Welch test in terms of worst-case size over a constrained variance-ratio range and achieves good calibration on partial biserial correlations, while showing higher power (and being much faster) on small-to-moderate samples than profile likelihood-ratio techniques.

Phil Assheton

statsadvice.com, Berlin, Germany.

Simulation-Based Inference Normalizing Flows Statistical Hypothesis Testing

Sep 1, 2026stat.ML

Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches

Conditional independence tests (CITs) test for conditional dependence between two random objects

X

and

Y

given a third random object

Z

. Existing CITs have limited applicability to high-dimensional data, especially multimodal data like text. However, we show that such tests are of interest for large language model (LLM) outputs, where we test whether an output

X

generated from a source text

Z

carries information about an attribute

Y

beyond

Z

itself. For this purpose, we propose embedded CITs (eCITs), which embed

X

and

Z

and apply an existing CIT to the resulting representations and to

Y

. We show that, provided the embedding of

Z

is sufficient, i.e. retains the information

Z

carries about either

Y

or the representation of

X

, the null hypothesis transfers from

X

and

Z

to their representations, so that a CIT valid for the embedded hypothesis is valid for the original one. We further give conditions for equivalence of the two hypotheses, and show that sufficiency weakens to mean sufficiency when the embedded test targets conditional mean independence. We propose a semi-synthetic simulation design to assess type I error (T1E) control and power of the eCITs for given embedding maps on a specific dataset and task, and use it to evaluate them on our application. Applying the eCITs to German Parliament speeches, we find for all combinations of embedding maps considered that the summaries of two LLMs contain information about the speaker's faction and gender beyond the speech they were generated from.

Marco Simnacher, Georg Keilbar, Benjamin König +2

Chair of Statistics Humboldt-Universität zu Berlin Spandauer Str. 1, 10178 Berlin, Germany · Chair Digital Health & Machine Learning Hasso-Plattner-Institut for Digital Engineering Prof.-Dr.-Helmert-Straße 2-3, 14482 Potsdam, Germany

LLM Evaluation Privacy Leakage in Language Models Conditional Independence Testing Statistical Hypothesis Testing

Aug 12, 2026cs.LG

Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough

Practitioners enrich customer-return models with ever more signals (lifetime value, category, recency/frequency, calendar, geography), and the temporal-point-process (TPP) literature follows suit with covariate- and external-covariate-conditioned intensities. But does any of it improve the timing, and how would you know? A null ("feature X doesn't help") is only meaningful if the model could have found a signal. We make two contributions--a method and a measurement--to answer this credibly. (i) A screen-and-confirm protocol that certifies whether a candidate signal improves a TPP's event-timing likelihood: a positive control plants a coupling of known strength and confirms the model recovers it, so a real-data null can be read as "no signal" rather than "weak method." The control is validated for categorical and continuous encodings, and on a real clock-driven dataset (NYC taxi hour-of-day). (ii) A model-free ceiling quantifying how little of customer-return timing is point-predictable at all (a single-digit percentage of gap variance from any covariate; returns are near-memoryless). With these we certify a clean result on three public benchmarks (Amazon, Taobao, RetailRocket) and a real marketplace (Thumbtack): the inter-event clock--continuous-time decay, long known to beat frozen-intensity models--is nearly sufficient, and the conditioning the field keeps adding is redundant or harmful on top of it (statistically null on the public benchmarks, at most 0.06 NLL; null to mildly harmful on the marketplace). We do not claim to discover that decay helps; our contribution is the tools that turn "conditioning doesn't help" into a checkable, certified statement--plus an honest-evaluation account of the read-out/leakage pitfalls we hit and retracted.

Sang Su Lee, Vineeth Loganathan, Shishir Dash +1

Thumbtack, Inc. San Francisco, CA, USA

Event Time Prediction Temporal Point Processes Statistical Hypothesis Testing

Aug 11, 2026stat.ML

Conditional Independence Tests for Constraint-Based Causal Discovery: A Survey

Conditional Independence (CI) tests are the statistical engine of constraint-based causal discovery: in algorithms such as PC (Peter-Clark) and FCI (Fast Causal Inference), skeleton pruning and key orientations follow directly from CI decisions. This survey reviews CI testing with emphasis on assumptions, robustness, and scalability in high-dimensional and mixed-type settings common in biomedical domains. The survey organizes widely used CI methods into six families: partial-correlation, contingency-table, regression, nearest-neighbor, kernel, and machine-learning-based. Special emphasis is provided on the robustness layers that address the limitations of these families. For each family, the survey examines when CI decisions reflect the data-generating distribution and when they fail. By this, we link test-level properties, including power decay with conditioning set size and asymmetric type I/II error consequences, to graph-level errors in skeleton recovery and v-structure orientation. The survey also compares adoption across major R and Python libraries and summarizes open challenges, including mixed-type CI testing without discretization, small-sample error control, and strategies for improving scalability of CI-testing.

Pavel Averin, Theodoros Moysiadis, Ioannis Katakis

Department of Computer Science School of Sciences and Engineering University of Nicosia 2417, Nicosia, Cyprus

Causal Discovery Causal Inference Conditional Independence Testing Statistical Hypothesis Testing

Aug 7, 2026cs.AI

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Jiacheng Miao, Jin Mu, Guanhua Chen +1

Department of Biomedical Data Science, Stanford University · Department of Genetics, Stanford University · Department of Biostatistics and Medical Informatics, University of Wisconsin–Madison +3

AI Agent Benchmarks LLM Agent Training RL for LLM Agents LLM Agent Reliability Statistical Hypothesis Testing

Aug 6, 2026cs.AI

NxN E-valuation: Hypothesis Certification via a Conformal CRT Null

We propose NxN E-valuation, a handy, e-value-based hypothesis-certification algorithm that lets a hypothesis be verified without building any case-specific certification procedure---such as constructing a dedicated null hypothesis---as long as a large enough dataset is available. The method is especially suited to LLM-based exploration systems, where LLMs are remarkably good at proposing hypotheses but suffer badly from hallucination; this hallucination prevents us from harvesting LLM outputs directly, and existing remedies each fall short. The most common solutions include letting the LLM verify or correct itself circular verification and held-out testing (where false hypotheses can still pass via spurious correlations), among other remedies detailed in the introduction. To resolve this, NxN E-valuation exploits the naturally existing large training set and lets different samples serve as null hypotheses for one another. This design directly realizes a conditional randomization test (CRT) that certifies each hypothesis. The approach can be a universally better replacement for at least LLM circular verification and held-out-data testing, provided the LLM's generations are hypotheses that apply to each individual sample.

Bin Wang, Yan Zhong, Liang Luo +2

Meta

LLM Hallucination Mitigation E-Values Statistical Hypothesis Testing

Aug 6, 2026cs.LG

Hypothesis Testing with Conditional Queries: Learnability and the Value of Interaction

Model evaluations may fix all tests before observing any responses or select later tests using earlier responses. We study this choice in a conditional-query model on a finite outcome space

\mathcal{X}

with

|\mathcal{X}|=N

. We first ask which pairs of distribution classes can be reliably distinguished. We then ask how many additional queries are required to match an adaptive tester when all queried events must be fixed in advance. We show that learnability holds if and only if the two classes have positive separation in their pairwise conditional probabilities. When this separation is zero, the optimal worst-case error is exactly

1/2

at every finite query budget. For any

T

-query adaptive policy and any

ρ\in (0,1)

, we construct a randomized non-adaptive procedure using

O(N^2(T + \log(1/ρ)))

pair queries chosen before any response is observed. Its simulated transcript is within

ρ

in total variation of the adaptive transcript, uniformly over all distributions in the model. We also construct a matching family with constant adaptive query complexity and

Ω_\varepsilon(N^2)

non-adaptive query complexity. Consequently, the worst-case fixed-error adaptivity gap is

Θ_\varepsilon(N^2)

. Thus interaction can reduce the required number of tests by a quadratic factor, but the apparent exponential branching of an interactive evaluation does not yield an exponential query advantage.

Zonghuan Xu

Fudan University

Statistical Hypothesis Testing

Aug 5, 2026stat.ME

Nonparametric Goodness-of-fit Testing under Covariate Shift

This paper develops procedures for nonparametric goodness-of-fit testing under covariate shift, where labelled data are drawn from a source population but goodness-of-fit is evaluated for a target population. The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio. Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function. The truncation stabilizes the importance- weighting kernel ridge regression as well as the bootstrap calibration, making our approach applicable even when the density ratio has heavy tails. We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions, and establish explicit error rates for coverage probability under specific conditions on the target- to-source density ratio and on the spectral decay of the kernel integral operator. Numerical experiments corroborate our theoretical findings.

Zhen Hou, Dong Xia

Department of Mathematics Hong Kong University of Science and Technology

Kernel Ridge Regression Covariate Shift Statistical Hypothesis Testing

Aug 5, 2026stat.ML

Automatic Statistical Test for Rationally Expressible Algorithms by Selective Inference, with Applications to Feature Selection

Selective inference (SI) provides statistically valid

p

-values for hypotheses selected by applying an algorithm to the data, correcting for the bias that arises when the same data are used both to select and to test a hypothesis. Developing an SI procedure for a new algorithm, however, has required an expert to derive, and then implement, the selection event, i.e., the conditions under which the hypothesis is selected. Repeating this specialized effort for every new algorithm is why exact SI has so far been available for only a narrow class. We propose AutoSI, a framework that removes this barrier in two ways. First, AutoSI constructs the selection event automatically from the algorithm's individual operations, so the user only writes the algorithm as ordinary NumPy-like code and derives nothing by hand. Second, AutoSI broadens the class of selection events SI can handle: existing exact methods are limited to selection events characterized by linear or quadratic inequalities in the data, whereas AutoSI covers any algorithm expressible through rational functions of the data (ratios of polynomials). We prove that the

p

-values computed by AutoSI are exactly valid in finite samples. We demonstrate AutoSI on three feature-selection methods, each written in a few dozen lines of code. One of these methods, the lasso with its tuning parameter selected by cross-validated

R^2

, cannot be handled within existing exact SI frameworks and is made possible by AutoSI. Experiments on synthetic and real datasets show that the resulting

p

-values control the type I error rate (i.e., the false positive rate) at the nominal level while retaining high power.

Teruyuki Katsuoka, Tomohiro Shiraishi, Shuichi Nishino +1

Nagoya University · RIKEN

Selective Inference Selection Bias Statistical Hypothesis Testing

Aug 4, 2026cs.CL

On the Non-Specificity of Statistical Measures Used in Script Decipherment

Statistical regularities are routinely offered as evidence that undeciphered sign systems encode language; the Indus script debate is the canonical example. Any such inference rests on specificity: the reported outcome must be unusual among plausible structured non-languages. We test that premise constructively with SIGIL, a purpose-built generative emblem system whose 3,000-text core corpus carries explicit compositional meanings although no sign has a phonological value. A literature registry compiled in advance of evaluation records 54 methods and admits a method to exact scoring when both the published Indus outcome and a source-defined decision rule can be reproduced. SIGIL receives the same category as the Indus corpus on every criterion scored this way, across repetition, directional-asymmetry, and lexical-distribution tests. Declared reconstructions of entropy, frequency, positional, predictive, classifier, and network measures reproduce the familiar Indus-like signatures as well. A sequential decipherment stress test then reaches high dictionary coverage for English, Sanskrit, and Tamil on the same corpus, while grouped held-out declines and unstable keys reveal how little that coverage identifies. The construction does not decide what the Indus signs encode: it shows that the evaluated measures detect organization without being specific to language, and therefore cannot, on their own, establish encoded speech.

Nikhil Raghavendra

Independent Researcher

Statistical Hypothesis Testing

Aug 3, 2026math.ST

Beyond Modern Asymptotics for Log-Likelihood Ratios in Logistic Regression

We characterize the finite sample behavior of the log-likelihood ratio statistic in binary logistic regression, uniformly over both the design and the target parameter. For

n\geq d\geq 3

, we determine, up to universal constants, its worst case

(1-δ)

quantile over all fixed collections of design vectors and all target parameters:

d\log\left(\frac{e n}{d}\right)+\log\left(\frac{1}δ\right).

This is a nonasymptotic analogue of the Wilks

χ^2_d

phenomenon and requires no regularity assumptions on the design. The low dimensional cases exhibit unusual behavior. The worst case quantile in dimension

d=2

is sharply of order

\log\log\log n+\log\left(\frac{1}δ\right).

The worst case quantile in dimension

d=1

is of order

\log(1/δ)

, with no dependence on

n

. Finally, i.i.d. Gaussian design vectors recover the classical Wilks scale. In the regime

n\gtrsim d+\log(1/δ)

, we prove the sharp bound

d+\log\left(\frac{1}δ\right).

Unlike existing asymptotic results, our bounds are uniform over the target parameter, which may depend on

n

,

d

, and

δ

.

Hugo Chardon, Reese Pathak, Nikita Zhivotovskiy

Department of Statistics, University of California, Berkeley · School of Operations Research and Information Engineering (ORIE), Cornell University

Logistic Regression Statistical Hypothesis Testing

Jul 23, 2026cs.LG

Zero-Flow Two-Sample Tests

Motivated by the success of modern flow-based generative models in modeling complex data, we study two-sample testing through the lens of flow-based methods. We propose the Zero-Flow Two-Sample Test (ZF2ST), built on the zero-flow criterion, which characterizes distributional equality through a time-reversal antisymmetry of a learnable velocity field. We extend this criterion and further develop the Zero-Flow Discrepancy, an identifying discrepancy that controls the Wasserstein distance, and derive a variational representation in terms of a witness function. This representation naturally leads to a witness-based test whose power is governed by the signal-to-noise ratio (SNR), allowing direct power maximization for witness learning. ZF2ST learns the witness on one data split and performs testing on held-out samples, thereby maintaining Type-I error control and admitting a simple asymptotic null distribution. Experimentally, ZF2ST performs competitively across various synthetic and real-world benchmarks, while showing particularly strong performance in distinguishing image distributions from different sources.

Yakun Wang, Leyang Wang, Song Liu +1

University of Bristol · RIKEN AIP · University of Tokyo & RIKEN AIP

Two-Sample Testing Statistical Hypothesis Testing

Jul 22, 2026stat.ML

Statistical Inference for Rank Allocation in Low-Rank Adaptation

Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.

Yihang Gao, Vincent Y. F. Tan

Department of Mathematics, National University of Singapore · Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Statistical Hypothesis Testing

Jul 17, 2026stat.ME

Aggregation of Statistical Evidence under Exchangeability

We study aggregation of statistical evidence under unknown and potentially complex dependence using group-invariance. Building on permutation-based constructions that treat transformed datasets as exchangeable units, we aggregate evidence across statistics for each transformed dataset and calibrate the resulting aggregates across transformations. We develop a finite-sample power and adaptivity theory for this framework, together with extensions to sequential and data-dependent aggregation that preserve validity. For single-batch aggregation, which uses one collection of transformed datasets for both standardization and calibration, we show that the critical values uniformly improve on deterministic calibrations valid under arbitrary dependence, including Bonferroni correction, while adapting to the unknown dependence structure. We also introduce a sequential alpha-spending version that permits early rejection when evidence is strong, and a two-batch extension that separates standardization from calibration to accommodate learned aggregation rules and reduce computation. Applications to adaptive nonparametric testing and conformal prediction illustrate how these results sharpen existing aggregation methods.

Antonin Schrab, Rajen Shah, Arthur Gretton +1

Department of Computer Science and Technology, University of Cambridge, UK · Statistical Laboratory, University of Cambridge, UK · Gatsby Computational Neuroscience Unit, University College London, UK +1

Sequential Hypothesis Testing Multiple Hypothesis Testing Statistical Hypothesis Testing

Jul 16, 2026math.ST

Measuring Spatial Clustering via Metropolis-Hastings Diffusion Distance

We propose a novel measure of the discrepancy between two probability distributions

f

and

g

on a graph - which we call the diffusion distance - that measures the rate of convergence of

f

to

g

under a graph-constrained Markov chain with stationary distribution

g

. As a default choice for this Markov chain, we use the Metropolis-Hastings transition matrix targeting

g

with proposals given by a random walk on the graph. Our primary case of interest is when the second distribution

g

is uniform, in which case the diffusion distance becomes a measure of spatial clustering in

f

. Used in this way, (Metropolis-Hastings) diffusion distance to uniformity extends Moran's

I

-type measures of spatial autocorrelation by incorporating global graph geometry rather than just local patterns. Indeed, Moran's

I

, the most well-known measure of spatial autocorrelation, can be viewed as a one-step heuristic for diffusion distance, so long as specific spatial weights are used. We establish theoretical bounds and a stability result for our measure, connecting it to graph spectra and optimal transport. We then turn our attention to outlining a statistical test for spatial clustering using diffusion distance. Under permutation null models, we derive high-probability bounds on diffusion distance underpinned by exact spectral formulas for convergence of distributions, enabling an efficient statistical test for spatial clustering on large datasets. We empirically compare diffusion distance to Moran's

I

both as a numerical measure and as a statistical test. We show that diffusion distance exhibits higher power on synthetic data using a stochastic block model. Empirical analysis of Black population distributions for 100 U.S. cities shows that diffusion distance detects subtle differences in urban segregation patterns that Moran's

I

does not.

Thomas Weighill, Chidinma Williams

Department of Mathematics and Statistics, University of North Carolina at Greensboro, Greensboro, NC 27402

Clustering Graph Diffusion Statistical Hypothesis Testing

Jul 13, 2026stat.ML

Falsifying Causal Graphs With Outlier Events

True causal relationships are rarely known, and inferring causal graphs from data is hard. A fundamental challenge is how to assess whether a given causal graph is good in the absence of a ground truth. We propose falsifying candidate causal graphs based on whether they can explain the propagation of an outlier event. Our approach leverages a key principle: weak outliers rarely cause strong ones. While this principle has previously been used in root cause analysis to identify root causes without prior knowledge of the graph, we turn it on its head and use it to falsify candidate causal graphs whose implied outlier propagation is inconsistent with the data. To this end, we present the first statistical tests for the hypothesis that a candidate graph is the true causal graph, and show they have false positive control, power guarantees against incorrect causal graphs, and can operate with a single outlier sample.

William Roy Orchard, Philipp M. Faller, Dominik Janzing

University of Cambridge, Cambridge, UK · Karlsruhe Institute of Technology, Karlsruhe, Germany · Amazon Research, Tübingen, Germany

Causal Discovery Statistical Hypothesis Testing

Jul 9, 2026cs.LG

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability - a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.

Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3

IBM Research

Early Stopping Optimal Stopping Automated Evaluation Sequential Hypothesis Testing Statistical Hypothesis Testing

Jul 5, 2026cs.CL

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

,

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Jun 18, 2026stat.ML

Betting on Moments: Legendre Jumper Martingales for Online Exchangeability Testing

A fundamental assumption in statistics and machine learning is that ``the future looks like the past,'' formalized as exchangeability: the joint data distribution is order-invariant. In practice, this assumption is often violated due to distribution shifts over time. Early detection of exchangeability violations is crucial to prevent performance degradation and enable timely interventions like model retraining. Conformal test martingales offer a flexible, distribution-free framework for sequential exchangeability testing with guaranteed false-alarm rate control by betting against the uniformity of conformal p-values. While alternatives such as plug-in martingales and mixture-based strategies exist, computationally efficient baselines like the Simple Jumper are limited to detecting mean location shifts. We propose a family of conformal test martingales based on shifted Legendre polynomials that extend the Simple Jumper to higher-order moments. The Simple Legendre Jumper replaces linear betting functions with polynomials of arbitrary degree, enabling rapid detection of variance, skewness, and other higher-order deviations. The Product Legendre Jumper combines multiple polynomial degrees into a single betting function but suffers from exponential state-space growth, termed the jumping tax. To resolve this, we introduce the Variational Legendre Jumper, which employs a mean-field approximation to reduce complexity to constant time per step with minimal power loss, providing an expressive, scalable framework for real-time distribution shift monitoring.

Johan Hallberg Szabadváry

Department of Mathematics, Stockholm University

Conformal Prediction Distribution Shift Detection Change-Point Detection Sequential Hypothesis Testing Temporal Distribution Shift Statistical Hypothesis Testing

Jun 17, 2026stat.ML

Sequential Kernel-based Conditional Independence Testing via Adaptive Betting

Testing conditional independence is fundamental yet intrinsically difficult: without additional assumptions, Type I error control is impossible in general. The "Model-X'' paradigm addresses this difficulty by assuming exact knowledge of a relevant conditional distribution. While small deviations from this assumption can sometimes be tolerated in classical one-shot testing, existing sequential conditional independence tests typically require the Model-X conditional to be known exactly, making them fragile when it must instead be estimated. We propose a new approach that is substantially more robust to such estimation error. Our method applies testing-by-betting to an adaptively optimized Kernel Conditional Independence statistic, together with a normalization scheme and a truncate-and-shift calibration strategy. These modifications greatly reduce Type I error inflation while preserving high power across high-dimensional synthetic benchmarks and real-world fairness tasks, outperforming existing sequential Model-X approaches. Code is available at https://github.com/he-zh/SKCI.

Zheng He, Danica J. Sutherland

Department of Computer Science, University of British Columbia, Vancouver, Canada · Alberta Machine Intelligence Institute, Edmonton, Canada

Kernel Methods Conditional Independence Testing Sequential Hypothesis Testing Statistical Hypothesis Testing

Jun 16, 2026stat.ML

Fast Nonparametric Conditional Independence Testing via Two-Stage Regression

Constraint-based causal discovery relies on repeated conditional independence tests, but fast nonparametric tests often sacrifice calibration, especially when variables depend on the conditioning set through nonlinear relationships. We introduce BLITZ (Broad-to-Local Independence Testing via residualiZation), a nonparametric conditional independence test designed to run well under a second while maintaining the accuracy needed for the thousands of queries performed by constraint-based causal discovery algorithms. BLITZ first removes broad smooth dependence on the conditioning set using low-order polynomial regression, then applies a small nonlinear feature map and residualizes those features with shallow tree regressions. The resulting statistic tests residual cross-covariance, with a moment-matched chi-square approximation to the null distribution. We show theoretically that the two-stage design reduces the effective complexity faced by the tree residualizers, allowing shallow trees to control residual conditional-mean bias while avoiding excessive overfitting. In simulations, BLITZ provides better null calibration than fast kernel, random-feature, and regression-based competitors while remaining among the fastest methods tested. In causal discovery experiments on synthetic graphs and flow-cytometry data, BLITZ yields more reliable endpoint orientations among retained adjacencies and competitive structural recovery. These results suggest that broad-to-local residualization is a practical route to calibrated, scalable nonparametric conditional independence testing for causal discovery.

Eric V. Strobl

Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15213, USA

Causal Discovery Causal Structure Learning Causal Inference Conditional Independence Testing Statistical Hypothesis Testing