The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering

Adam Noonan

stat.MLAug 21, 2026

The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering

Authors: Adam Noonan

Organizations: Independent Researcher

Abstract

Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by $1+(m-1)ρ_I(p)$ , where $m$ is the group size, $p$ is the target fraction, and $ρ_I(p)$ measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations' worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the dataset changed. We asked it a different question. The number of rows is a property of the dataset. The effective sample size belongs to the analysis.

Figures & tables

Mixture weight $q$	Score correlation $\rho$	Indicator correlation $\rho_{I}(0.9)$	Indicator design effect	Score-based substitute
0.75	0.50	0.7222	1.7222	1.50
0.50	0.00	0.4444	1.4444	1.00
0.30	$-0.40$	0.2222	1.2222	0.60

Table 1

Score correlation $\rho$	Indicator correlation at $p_{n}$	Effective size	Simulated sd	First-order sd	Score-based sd
0.00	0.0000	200.0	0.0212	0.0212	0.0212
0.20	0.0798	161.4	0.0234	0.0236	0.0268
0.40	0.1847	128.7	0.0265	0.0264	0.0314
0.60	0.3221	101.7	0.0299	0.0297	0.0354
0.80	0.5135	78.7	0.0341	0.0337	0.0390
0.95	0.7545	61.3	0.0388	0.0382	0.0415

Table 2

Figure 1: Gaussian simulation and coverage approximations. Left: simulated coverage standard deviations compared with indicator- and score-based predictions. Right: fitted Beta approximations at three dependence levels. The Beta shapes are approximations, not a conclusion of Theorem 1.

Figure 2: Level dependence under a Gaussian copula. Left: indicator correlation at selected levels compared with score correlation. Right: the resulting design effect. The ordering and decline shown here belong to this family; Section 3.1 gives a different ordering.

Quantity	Measurement
Rows / question clusters	25,028 / 500
Cluster size: minimum / median / maximum	8 / 44 / 135
Mean / size-biased mean cluster size	50.06 / 61.29
Raw-score ICC	0.599
Indicator ICC at the reported proxy level	0.495
Plug-in design effect	30.8

Table 5

Level $p$	Indicator correlation	Design effect	Sentence draws	Document draws [95% band]	Ratio
0.50	0.135	4.42	1,002	217 [208, 227]	4.62×
0.80	0.083	3.09	1,014	326 [312, 341]	3.11×
0.90	0.037	1.95	1,019	485 [465, 507]	2.10×
0.95	0.027	1.69	1,000	621 [595, 649]	1.61×

Table 6

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Analyte	Score ICC	$\hat{\rho}_{I}(0.90)$	$\hat{\rho}_{I}(0.95)$	$\hat{\rho}_{I}(0.975)$	$\hat{\rho}_{I}(0.99)$
Sodium	0.236	0.1133	0.0732	0.0732	0.0364
Potassium	0.104	0.0449	0.0215	0.0149	0.0074
Albumin	0.119	0.0307	0.0241	0.0128	0.0049
Creatinine	0.036	0.0136	0.0050	0.0036	0.0032
Bilirubin	0.114	0.0133	0.0042	0.0046	0.0023

Appendix

Table 7

$b$	Mean drift $\mathbb{E}C-p_{n}$	$b$ times drift	Candidate smooth coefficient, multiplied by $b/n$
25	$-0.006793$	$-0.1698$	$-0.1724$
50	$-0.003418$	$-0.1709$	$-0.1723$
100	$-0.001715$	$-0.1715$	$-0.1722$
200	$-0.000859$	$-0.1718$	$-0.1722$
400	$-0.000430$	$-0.1720$	$-0.1722$

Appendix

Table 8

Size profile	Mean size	Size-biased mean	Simulated sd	Predicted sd: size-biased mean	Predicted sd: plain mean
Equal, size 4	4.00	4.00	0.0339	0.0335	0.0335
Sizes 3, 4, 5	3.98	4.15	0.0344	0.0341	0.0336
Sizes 1, 2, 4, 9	4.00	6.38	0.0397	0.0391	0.0322
Sizes 1, 2, 4, 8, 16	3.76	8.53	0.0467	0.0466	0.0330

Appendix

Table 9

Choice of $n_{e}$	Effective size	KS distance
Nominal row count	25,028	0.3128
Indicator-ICC plug-in	812	0.0710
Matched to the bootstrap variance	1,290	0.0170

Appendix

Table 10

Result	Source scripts
Gaussian dispersion comparison	verify_indicator_icc.py , sim_validation.py
Score-correlation counterexample and dependence checks	assumption_stress.py
Tail-limit calculations	verify_tail_limit.py , evt_tail_rate.py
Unequal-size and ICC estimation examples	ragged_and_estimation.py , icc_estimators.py
Released proxy measurements and resampling	prm_measurement.py , prm_dispersion.py , test_marginal_scope.py
SQuAD score–indicator comparison (Section 3.3)	score_squad2.py , measure.py , d01_measure.json

Appendix

Table 11

Explore similar work

Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
Sep 23, 2026Spandan Ghose ChowdhuryHeterogeneous Treatment Effects Illusions
Georgia Institute of Technology, GA, USA
Intrinsic effective sample size for manifold-valued Markov chain Monte Carlo via kernel discrepancy
May 5, 2026Kisung YouMarkov Chain Monte Carlo Sample Size
Department of Mathematics, Baruch College · Department of Mathematics, The Graduate Center, City University of New York
ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations
Sep 29, 2026Anthony Lavertu, Jacob Cote, Sophie Gobeil +3Relational Deep Learning Training-Inference Mismatch
Department of Computer Science Université Laval Québec, QC, Canada · Department of Biochemistry, Microbiology and Bioinformatics Université Laval Québec, QC, Canada · Department of Molecular Medicine Université Laval Québec, QC, Canada

Sep 23, 2026, stat.MEJ/K move · Enter open · S save

Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS

Spandan Ghose Chowdhury

Georgia Institute of Technology, GA, USA

Heterogeneous Treatment Effects Illusions

Open paper AskPDF arXiv

Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9837 parameter conditions (approx 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known, and partially when rho must be estimated from short pre-periods, but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias, though with a fixed instrument more data does sharpen the first stage, so identification rests on instrument strength, not sample size.

Abstract

Explore similar work

Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS

Intrinsic effective sample size for manifold-valued Markov chain Monte Carlo via kernel discrepancy

ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations