Statistical Power Analysis

Follow this topic Ask about this topic

Statistical power analysis studies how likely a statistical test is to detect an effect, helping researchers plan sample sizes and interpret null or uncertain results. Common approaches use formulas based on effect size, variance, and test design, while simulations handle complex models, clustering, attrition, and paired comparisons. In machine learning, these tools assess whether benchmark score differences or model rankings are reliably detectable, often using item-level data and minimum detectable effects. A central challenge is that power estimates depend on realistic assumptions about data, variability, and analysis choices, so a nominal sample-size target may not ensure a trustworthy conclusion.

Latest papers

13 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Equivalence testing and run-count design for stochastic machine-learning experiments
Oct 16, 2026Dmytro Knopov, Nadiya ShvaiStatistical Hypothesis Testing Statistical Power Analysis
Faculty of Computer Sciences, Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine. · Cyclope.ai, Paris, France.
Two-Sample Testing for Random Graphs without Vertex Correspondence
Oct 5, 2026Soham DanTwo-Sample Testing Statistical Power Analysis
Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
Sep 23, 2026Spandan Ghose ChowdhuryCausal Effect Estimation Statistical Power Analysis
Georgia Institute of Technology, GA, USA
A Statistical Approach to Estimating Sample Size of Machine Learning Models
Sep 9, 2026Dat Phan-Trong, Sunil Gupta, Svetha VenkateshStatistical Power Analysis
Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia
When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits
Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power Analysis Benchmark Auditing
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University
Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking
Jul 6, 2026Xiaopu Wang, Zelin He, Chengyuan Liu +1Statistical Power Analysis LLM Watermarking
Department of Statistics, Pennsylvania State University
evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
Jul 5, 2026Shreyas K ChandrahasLLM Evaluation Pairwise Comparison
Resolution Diagnostics for Paired LLM Evaluation
May 28, 2026Anany KotawalaLLM Evaluation Pairwise Comparison
Princeton University, Princeton, NJ, USA.
Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit
May 25, 2026Zexin Zhuang, Yanhang Li, Zhichao FanLLM Quantization Statistical Power Analysis
1Southern Methodist University · 2Northeastern University · University of Illinois Urbana-Champaign
Unstable Rankings in Bayesian Deep Learning Evaluation
Apr 25, 2026Qishi Zhan, Minxuan Hu, Guansu Wang +2Bayesian Neural Networks Statistical Power Analysis
Department of Mathematical and Statistical Sciences Marquette University, USA · Cornell University, USA · The University of Melbourne, Australia +2
Power Analysis for Prediction-Powered Inference
Mar 17, 2026Yiqun T. Chen, Moran Guo, Shengy LiStatistical Power Analysis Prediction-Powered Inference
Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. · Department of Computer Science, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A.
Sample Size Calculations for Developing Clinical Prediction Models: Overview and pmsims R package
Feb 26, 2026Diana Shamsutdinova, Felix Zimmer, Oyebayo Ridwan Olaniran +4Learning Curves Clinical Prediction
Department of Biostatistics and Health Informatics, King’s College London · NIHR Biomedical Research Centre, Maudsley NHS Trust
Measuring all the noises of LLM Evals
Dec 24, 2025Sida I. WangLLM Evaluation Automated Evaluation
FAIR at Meta

Oct 16, 2026J/K move · Enter open · S save

Equivalence testing and run-count design for stochastic machine-learning experiments

Dmytro Knopov, Nadiya Shvai

Faculty of Computer Sciences, Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine. · Cyclope.ai, Paris, France.

Statistical Hypothesis Testing Statistical Power Analysis ML Reproducibility

Open paper AskPDF arXiv

Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; its shifted-t refinement is calibrated against the realised two-level process. Simulations on a 270-cell grid and under non-Gaussian seed laws confirm the formula within its design-relevant range, from five runs per method. For four published runs of a joint energy-based model, the failure to certify equivalence was largely decided by design: with the variance calibrated to those runs, two per method certify with probability near three per cent, whereas certifying 0.01 AUROC at 80 per cent power needs twelve even under exact equality. A second case study on a thousand-seed CIFAR-10 zoo and the MultiBERTs checks the formula against real seed laws (within 0.05 in that range) and resolves comparisons that two or three runs leave undecided. In a two-stage design (pilot, then formula), our experiments suggest that an upper confidence bound on the pilot variance substantially reduces the undersizing seen with its point estimate.