Certifying Compressed Language Models: An Audit and a Statistical Toolkit

Amogh Singh

cs.LGAug 15, 2026

Certifying Compressed Language Models: An Audit and a Statistical Toolkit

Authors: Amogh Singh

Organizations: Georgia Institute of Technology

Abstract

A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original. That quantity is least informative when two models are most alike: a net delta is what survives cancellation between opposing per-item changes, and cancellation is most complete in the regime equivalence claims occupy. Across an atlas of 1,707 paired model-by-task cells mined from public per-item evaluation dumps (1.3B-405B), churn runs roughly five times the net accuracy delta, and cells scoring identically to their baseline still disagree on individual items. In a preregistered audit of 17 equivalence claims from three registered frames (method papers, model cards, vendor documentation), 16 are eligible. None states a prospective numerical equivalence margin, and none releases task-matched per-item outputs, though 3 release outputs for other tasks only; 5 report too little to assess numerically, so a reader cannot check them at any sample size. We audit evidential sufficiency, not truth: no claim is called false. We supply the missing instrument: paired equivalence testing at a declared margin, with certification tables giving the items an evaluation needs, computed from disagreement observed under compression, not from independent-binomial variance. A controlled experiment pairs GPTQ and AWQ on byte-identical calibration samples across five seeds. Under the frozen eight-cell decision rule H3 is supported: changing the calibration draw was sufficient to reverse the observed method ordering in 5 of 8 confirmatory cells. The reporting standard we propose is five lines: declare a margin, run the paired test, report churn beside net delta, cite the sample size you met, release per-item outputs. It applies to any comparison between two models alike enough to be worth comparing. All per-item outputs, protocols and code are released.

Figures & tables

Panel of Figure 1 — Figure 1: Near-equal aggregate accuracy does not certify interchangeable behavior. One registered cell: Qwen2.5-7B on GSM8K, 4-bit GPTQ against 4-bit AWQ, paired on byte-identical calibration samples across five seeds. (A) and (B) The aggregate gap of 0.58 pp is the small difference between two large opposing quantities summing to 17.66%. (C) The item count needed to certify equivalence at a declared $\pm 2$ pp margin is a planning requirement computed at an assumed true difference of zero: it says the evaluation cannot support the claim, not that the methods differ. (D) The sign changes with the calibration draw alone, and across the eight registered cells the winner reverses in 5 of 8. Scope. This cell is an illustrative example chosen because it is legible, and it is the most extreme of the eight: its churn-to-net-delta ratio is $30.4\times$ against a median of $12.7\times$ across the eight. All eight are shown in (D) for that reason. The atlas figures in the panel above are observational and describe the public evaluation record rather than a census of compression.

Panel of Figure 2 — Figure 2: The same evaluation, before and after the reporting standard. Both columns describe one registered cell: Qwen2.5-7B on GSM8K, 4-bit GPTQ against 4-bit AWQ, $1{,}000$ paired items. A is the aggregate report (two accuracies and a net delta of $0.58$ pp), and it is complete: there is no further evidence behind it. B is the same run reported under the five lines of § 10 , and it needs no additional experiment. Row 2 draws the declared $\pm 2$ pp margin together with the observed paired delta and the committed 95% percentile-bootstrap interval for it; no TOST verdict exists for this cell , and that interval is not the 90% two-sided interval TOST at one-sided $\alpha=.05$ requires (§ 3.2 ), so it settles the equivalence question in neither direction. Row 4’s $2{,}730$ is a planning requirement computed at an assumed true difference of zero: it says this evaluation cannot support an equivalence claim at $\pm 2$ pp, and it is not evidence that the two methods differ. Figure 1 decomposes the same cell; every value here is read from the artifacts recorded in paper/figures/fig2_values.json .

Benchmark	Atlas cells	Discordance p25/med/p75	Required $n$ p25/ med /p75	Naive	Advantage
musr	24	0.015 / 0.034 / 0.044	232 / 519 / 681	7,617	$14.7\times$
gpqa	24	0.035 / 0.048 / 0.067	545 / 749 / 1,035	7,233	$9.7\times$
bbh	192	0.024 / 0.044 / 0.060	371 / 681 / 928	6,492	$9.5\times$
mmlu_pro	5	0.048 / 0.053 / 0.059	739 / 827 / 913	7,695	$9.3\times$
hellaswag	23	0.031 / 0.045 / 0.099	477 / 695 / 1,538	4,905	$7.1\times$
arc_challenge	17	0.072 / 0.079 / 0.099	1,112 / 1,218 / 1,536	7,363	$6.0\times$

Table 1: Items required to certify equivalence within

\pm 2

pp under TOST at one-sided

\alpha=.05

(a 90% two-sided interval) with 80% power, assuming a true difference of zero, by benchmark family, at the 25th/50th/75th percentiles of the discordance rates the atlas observes for that family. The naive column is the same requirement computed by treating the two runs as independent samples. Eleven benchmark families plus the pooled row.

Panel of Figure 3 — Figure 3: How many items an equivalence claim needs, by benchmark family. The same data as Table 1 , drawn for reference at the registered $\pm 2$ pp margin: TOST at one-sided $\alpha=.05$ with 80% power, at an assumed true difference of zero. Each row is one family; the band spans the requirement at the 25th and 75th percentiles of the discordance the atlas observes for that family and the disc is the median. Families are ordered by that median, and the two right-hand columns are what the ordering does and does not follow: it tracks median churn and not median baseline accuracy. Other margins. The requirement scales as $1/m^{2}$ , so on this log axis a change of margin translates the whole chart without reshaping it: across all twelve rows the $\pm 1$ pp requirement is $4.00$ – $4.00\times$ the $\pm 2$ pp one and the $\pm 3$ pp requirement is $0.44$ – $0.45\times$ it. The ruler row marks the pooled median at all three margins on this same axis ( $825$ , $1{,}855$ , $7{,}420$ items). Scope. These are planning sizes, and a lower bound under any non-zero true difference (§ 3.2 ); meeting one makes the equivalence test informative, it does not certify anything on its own. Families with fewer than four analysable atlas cells are absent, and two of the rows shown rest on thin evidence (§ 3.6 ).

	S1	S2
Cells	1,398	309
Median accuracy-state churn	0.137452	0.048000
Median $\|$ net accuracy delta $\|$	0.026316	0.009242
TOST-equivalent at 2 pp	68 (4.9%)	53 (17.2%)
Exact McNemar $p<0.05$	371 (26.5%)	19 (6.1%)

Table 2: The two atlas strata, on the 1,707-cell analysis population. S1 is the Open LLM Leaderboard v1 archive of community quantizations of 2023-era base models, TheBloke-era GPTQ and similar; S2 is the Neural Magic / Red Hat W4A16, INT8 and FP8 releases from 8B to 405B. Percentages are within-stratum shares of cells. The two medians are given to six places because the ratios quoted in the text divide them unrounded; dividing the printed cells reproduces 5.22 and 5.19.

	Prospective numerical decision margin	Retrospective numerical description of a result	Qualitative language only, no number
Numerically assessable	0	5	7
Not numerically assessable	0	1	3
Total	0	6	10

Table 3: What the 16 eligible sources declare. Columns are the form of the evidence offered; rows are whether the source reports enough for a numerical assessment under the registered paired-outcome framework. Prospective numerical decision margin means a tolerance stated independently of the observed result: a threshold that could have been written down before the evaluation ran. That column is empty.

Classification across the atlas-IQR interval	Claims
Above the planning threshold throughout	10
Changes classification within the interval	1
Below the planning threshold throughout	0
Total assessable	11

Table 4: Sensitivity classification of the 11 numerically assessable claims at the registered 2 pp margin. Each claim’s required sample size is recomputed at the first and third quartiles of the atlas cells that supplied its median discordance; because required

n

increases monotonically in the discordance rate, the quartiles bracket the whole interval. Per-claim values are in Appendix C .

Model	Task	$\mathrm{gap}$	$\mathrm{range}_{\mathrm{GPTQ}}$	$\mathrm{range}_{\mathrm{AWQ}}$	Flip	Range/gap
Qwen2.5-1.5B	MMLU	0.012292	0.040521	0.018445	TRUE	TRUE
Qwen2.5-1.5B	GSM8K	0.096800	0.014000	0.033000	FALSE	FALSE
Llama-3.2-3B	MMLU	0.030922	0.063809	0.029341	FALSE	TRUE
Llama-3.2-3B	GSM8K	0.017800	0.011000	0.034000	TRUE	TRUE
Qwen2.5-7B	MMLU	0.006267	0.015667	0.012391	FALSE	TRUE
Qwen2.5-7B	GSM8K	0.005800	0.048000	0.035000	TRUE	TRUE

Table 5: The eight registered confirmatory cells.

\mathrm{gap}

is the absolute difference of the two five-seed mean accuracies;

\mathrm{range}_{m}

is the seed-wise spread of method

m

; the range/gap criterion holds when

\max(\mathrm{range}_{\mathrm{GPTQ}},\mathrm{range}_{\mathrm{AWQ}})\geq\mathrm{gap}

. Values reproduce the signed decision record.

	Atlas (observational)	Controlled
Ratio of medians	5.40	12.1
Median of per-cell ratios	3.85	12.7
Cells entering the per-cell median	1,562 of 1,707	8 of 8

Table 6: The churn-to-net-delta ratio under both aggregations, in both regimes. The two aggregations answer different questions, so the paper reports both rather than choosing: a ratio of medians describes the typical churn against the typical net delta, and a median of per-cell ratios describes the typical cell. The controlled regime exceeds the observational one either way. All figures here are descriptive summaries of the two populations; the comparison is not a registered hypothesis and no inferential test is performed on it. The last row is why the two columns are not exactly parallel: a cellwise ratio is undefined where the net delta is exactly zero, which never happens in the eight controlled cells and happens in 145 atlas cells, 6.8% of S1 and 16.2% of S2 (§ 4.3 ).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Date	Artifact	Commit	What it fixed in advance
2026-07-11	PREREGISTRATION.md	—	Grid, metrics, TOST margin (2 pp), H3 decision rule
2026-07-15	Three registrations	b74fd58	Audit, atlas-mining, and mini-grid protocols
2026-07-15	Audit Amendment 1	d6e02dd	Blind-extraction independence mechanism
2026-07-15	Pair manifest	f06348f	The 59 atlas pairs, before any flip statistic
2026-07-15	Claim table	715a7ce	The 17 audited claims, before any verdict
2026-07-20	WikiText-2 amendment	—	“Document” definition, Decision Point A

Appendix

Table 7: Freeze timeline. Each artifact was committed before the analysis it governs could be run, and each frozen file carries a Dated Amendments section: deviations are appended, never edited in, and each records whether results had been inspected before the decision.

Claim	Frame	Source
R01	F1	GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar, Ashkboos, Hoefler, Alistarh)
R02	F1	LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers, Lewis, Belkada, Zettlemoyer)
R03	F1	SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (Xiao, Lin, Seznec, Wu, Demouth, Han)
R04	F1	AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin, Tang, Tang, Yang, Chen, Wang, Xiao, Dang, Gan, Han)
R05	F1	SqueezeLLM: Dense-and-Sparse Quantization (Kim, Hooper, Gholami, Dong, Li, Shen, Mahoney, Keutzer)
R06	F1	A Simple and Effective Pruning Approach for Large Language Models [Wanda] (Sun, Liu, Bair, Kolter)

Appendix

Table 8: The 17 audited claims: identity and frame. F1 is a research-paper claim, F2 a vendor model card or official documentation page, F3 a vendor blog post. Source URLs, version dates and the exact quoted sentence per claim are in the frozen claim table docs/audit_claim_table.csv , which is part of the released artifact.

Claim	Family	Bits	Benchmark	$n$
R01	gptq	4	piqa	1,838
R02	w8a8_int8	8	(mixed/unmatched)	—
R03	w8a8_int8	8	(mixed/unmatched)	18,300
R04	awq	4	gsm8k	1,319
R05	squeezellm	4	mmlu	14,042
R06	pruning	—	(mixed/unmatched)	18,904

Appendix

Table 9: Per-claim characterisation and sensitivity classification, at the registered 2 pp margin.

n

is the reported or imputed item count, with its basis recorded in the released CSV.

p_{d}

is the discordance imputed from the atlas and tier the match tier it came from. V3 is whether the source releases per-item outputs for the tasks its claim rests on. Classification is evaluated across the interquartile range of the atlas cells supplying

p_{d}

: above throughout and below throughout hold at both quartiles, and sensitive changes within the interval. R10 keeps its row and is marked ineligible rather than removed.

Claim	$n$	MDD (paired)	MDD (indep.)	$n_{\mathrm{req}}$ @Q1	@median	@Q3	$d^{*}$
R01	1,838	2.36	3.62	1,364	2,010	4,328	0.1189
R02	—	—	—	619	866	1,237	—
R03	18,300	0.49	1.38	619	866	1,237	—
R04	1,319	2.09	3.77	1,137	1,137	1,137	—
R05	14,042	0.90	1.63	1,525	2,255	3,865	0.9085
R06	18,904	0.70	1.36	932	1,841	6,046	—

Appendix

Table 10: Power quantities per claim at the registered 2 pp margin. MDD is the minimum detectable difference in percentage points, under the paired and the independent-binomial variance models. Required

n

is the item count needed under TOST at one-sided

\alpha=.05

(a 90% two-sided confidence interval) with 80% power, assuming a true difference of zero, under the paired model; it is given at the first quartile, median and third quartile of the atlas cells supplying the claim’s discordance rate, which bracket the interval because required

n

increases monotonically in that rate.

d^{*}

is the discordance rate at which the classification would reverse; it falls inside the swept interval for R01 alone, and is blank where no reversal is attainable. Numbers on ineligible and non-assessable rows are not verdicts ; they are retained so a reader can see what the analysis could and could not evaluate.

Locus of the claim	Form it takes in the card	Cards
Prose assertion	states recovery percentages outright	3
Prose juxtaposition	states two scores, characterises neither	2
Table cell only	no comparative sentence anywhere	1

Appendix

Table 11: Where the equivalence claim is written, across six model cards from one publisher with the underlying evidence held constant. Every card reports the same recovery column; the tiers differ only in whether, and where, the card states the comparison in words. Read by § 5.3 .

	MDD		MDD / 2 pp
Claim	paired	indep. binomial	paired	indep. binomial
R01	2.36 pp	3.62 pp	$1.18\times$	$1.81\times$
R05	0.90 pp	1.63 pp	$0.45\times$	$0.82\times$
R07	0.87 pp	1.63 pp	$0.43\times$	$0.81\times$
R06	0.70 pp	1.36 pp	$0.35\times$	$0.68\times$
R17	0.61 pp	1.09 pp	$0.30\times$	$0.54\times$

Appendix

Table 12: Detection resolution (V1) at the registered 2 pp margin: the smallest difference each evaluation could have detected, and its ratio to that margin. A ratio below

1

means the evaluation resolves differences finer than the margin. The independent-binomial columns are the same quantities computed without the paired-design benefit. Rows are the 11 assessable claims; the two italicised rows carry no verdict and are shown only for transparency. The MDD columns repeat those of Table 10 ; the ratio columns appear only here.

Benchmark	Atlas cells	Discordance p25/med/p75	Required $n$ p25/ med /p75	Naive	Advantage
musr	24	0.015 / 0.034 / 0.044	928 / 2,076 / 2,721	30,468	$14.7\times$
gpqa	24	0.035 / 0.048 / 0.067	2,180 / 2,993 / 4,138	28,931	$9.7\times$
bbh	192	0.024 / 0.044 / 0.060	1,484 / 2,721 / 3,710	25,967	$9.5\times$
mmlu_pro	5	0.048 / 0.053 / 0.059	2,955 / 3,305 / 3,649	30,778	$9.3\times$
hellaswag	23	0.031 / 0.045 / 0.099	1,905 / 2,777 / 6,151	19,618	$7.1\times$
arc_challenge	17	0.072 / 0.079 / 0.099	4,447 / 4,870 / 6,141	29,451	$6.0\times$

Appendix

Table 13: Items required to certify equivalence within

\pm 1

pp under TOST at one-sided

\alpha=.05

Benchmark	Atlas cells	Discordance p25/med/p75	Required $n$ p25/ med /p75	Naive	Advantage
musr	24	0.015 / 0.034 / 0.044	104 / 231 / 303	3,386	$14.7\times$
gpqa	24	0.035 / 0.048 / 0.067	243 / 333 / 460	3,215	$9.7\times$
bbh	192	0.024 / 0.044 / 0.060	165 / 303 / 413	2,886	$9.5\times$
mmlu_pro	5	0.048 / 0.053 / 0.059	329 / 368 / 406	3,420	$9.3\times$
hellaswag	23	0.031 / 0.045 / 0.099	212 / 309 / 684	2,180	$7.1\times$
arc_challenge	17	0.072 / 0.079 / 0.099	495 / 542 / 683	3,273	$6.0\times$

Appendix

Table 14: Items required to certify equivalence within

\pm 3

pp under TOST at one-sided

\alpha=.05

Field	Value
pair_index / source	35 / S1
Task	harness_hendrycksTest_high_school_geography_5 (MMLU)
Base model	project-baize/baize-v2-7b
Quantized model	TheBloke/Project-Baize-v2-7B-GPTQ (GPTQ)
$n$	198
Baseline accuracy	0.429293

Appendix

Table 15: The most extreme zero-delta cell in the atlas: identical accuracy, symmetric flips,

p=1.0

. Read by § 4.3 , which carries the caveats that travel with it.

Cell	Winner flip	$\mathrm{gap}$	$\max\mathrm{range}$	Range/gap holds
Qwen2.5-1.5B / MMLU	TRUE	0.012292	0.040521	TRUE
Qwen2.5-1.5B / GSM8K	FALSE	0.096800	0.033000	FALSE
Llama-3.2-3B / MMLU	FALSE	0.030922	0.063809	TRUE
Llama-3.2-3B / GSM8K	TRUE	0.017800	0.034000	TRUE
Winner flip in 2 of 4 cells (threshold $\geq 1$ : met)
Range/gap holds in 3 of 4 cells (threshold $\geq 2$ : met)

Appendix

Table 16: The mechanical escalation screen over the four completed mini-grid cells. Outcomes are reproduced verbatim from the signed decision record; the per-seed accuracies that produce them are not restated here. “Winner flip” and “range/gap holds” are the two §3 predicates; the counts beneath drive the decision.

Model	Task	$d_{0}$	$d_{1}$	$d_{2}$	$d_{3}$	$d_{4}$
Qwen2.5-1.5B	MMLU	$-0.029127$	$-0.014029$	$+0.012463$	$-0.027631$	$-0.003133$
Qwen2.5-1.5B	GSM8K	$-0.111000$	$-0.098000$	$-0.080000$	$-0.088000$	$-0.107000$
Llama-3.2-3B	MMLU	$-0.008973$	$-0.090514$	$-0.023287$	$-0.015026$	$-0.016807$
Llama-3.2-3B	GSM8K	$+0.021000$	$+0.015000$	$-0.005000$	$+0.023000$	$+0.035000$
Qwen2.5-7B	MMLU	$+0.006694$	$+0.009400$	$+0.001638$	$+0.001496$	$+0.012107$
Qwen2.5-7B	GSM8K	$+0.022000$	$+0.016000$	$-0.023000$	$+0.002000$	$+0.012000$

Appendix

Table 17: Per-seed differences

d_{s}=\mathrm{acc}_{\mathrm{GPTQ},s}-\mathrm{acc}_{\mathrm{AWQ},s}

on byte-identical calibration samples. A winner flip is two seeds of opposite, nonzero sign; the five cells with a flip are those whose row is not sign-constant. No entry is zero, which is the 0-of-40 tie count stated in § 6.2 .

		Seed-level SD		Item-level SE
Model	Task	GPTQ	AWQ	GPTQ	AWQ
Qwen2.5-1.5B	MMLU	0.019130	0.006897	0.004215	0.004212
Qwen2.5-1.5B	GSM8K	0.006340	0.015116	0.015796	0.015658
Llama-3.2-3B	MMLU	0.024925	0.011157	0.004214	0.004216
Llama-3.2-3B	GSM8K	0.004615	0.012300	0.015154	0.015308
Qwen2.5-7B	MMLU	0.006224	0.004537	0.003983	0.004002

Appendix

Table 18: The three registered supporting analyses over the eight confirmatory cells. (a) Variance components, reported separately as registered: SD is across the five calibration seeds, SE is the item-level standard error within a cell, and MMLU cells run at

n=14{,}042

against GSM8K’s

n=1{,}000

, which is what drives the SE columns. (b) Two-level paired bootstrap, 2,000 replicates at RNG seed 0, both registered; tie replicates are reported separately and are included in the rank-flip denominator, per the registered convention, and the final column repeats the registered winner-flip outcome, which is a different question and not a consistency check. (c) Flip statistics, GPTQ against AWQ, means over the five paired seeds: net delta is

\mathrm{acc}_{\mathrm{AWQ}}-\mathrm{acc}_{\mathrm{GPTQ}}

, harmful and beneficial are the two directions of correctness change, accuracy-state churn is their sum, and total answer churn adds items that change answer without changing correctness state.

Task	$\max\mathrm{range}$ in paired SE	$\mathrm{gap}$ in paired SE
MMLU ( $n=14{,}042$ )	5.53 – 17.45	2.21 – 8.45
GSM8K ( $n=1{,}000$ )	1.92 – 3.61	0.44 – 5.65

Appendix

Table 19: Post-hoc resolution analysis, read by § 6.5 . Ranges of the two ratios across the four cells of each task.

p_{d}

is computed per cell from panel (c) of Table 18 and is not the harness study’s

\bar{Q}

, which is a quantized-against-FP16 contrast on a different pair. This quantity was not registered; it was requested after the eight-cell verdict was computed and signed, it tests no hypothesis, and it modifies no verdict.

Cond	Configuration vs. reference	Acc	Net $\Delta$	$C_{\mathrm{cond}}$	c $\to$ i / i $\to$ c	$R=C_{\mathrm{cond}}/\bar{Q}$
GSM8K , $n=200$ , reference accuracy $0.575$ , $\bar{Q}=0.287$
A	exemplars as separate turns	0.515	$-0.060$	0.240	30 / 18	0.836 $=0.240/0.287$
B	strict-match rescore	0.120	$-0.455$	0.455	91 / 0	1.585 $=0.455/0.287$
C	chat template off, 3-shot	0.475	$-0.100$	0.320	42 / 22	1.115 $=0.320/0.287$
D	stock defaults (chat off, 5-shot)	0.495	$-0.080$	0.300	38 / 22	1.045 $=0.300/0.287$
MMLU , $n=400$ , reference accuracy $0.415$ , $\bar{Q}=0.199$

Appendix

Table 20: Configuration sensitivity on FP16 Qwen2.5-1.5B, against the quantization denominator

\bar{Q}

for each task.

C_{\mathrm{cond}}

is correctness-state churn versus the reference;

R=C_{\mathrm{cond}}/\bar{Q}

is reported only with both inputs in view. The

\text{c}{\to}\text{i}/\text{i}{\to}\text{c}

column splits churned items by direction so churn is never read as net degradation. Conditions C and D are byte-identical for MMLU and are reported once as

C\equiv D

(registration §3.3).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Oct 20, 2026

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

Model compression makes large language models deployable, but today's evaluation tells an incomplete story. Standard practice measures retained task performance and stops there. This misses a critical fact: compression perturbs the distributed knowledge embedded across billions of weights, and task scores alone cannot reveal which underlying abilities survive and which quietly collapse. To solve this question, we introduce ACE-Compression, an ability-centric evaluation framework. Instead of asking only "does the compressed model still answer correctly?", we ask "which of its core abilities remain intact?" Our framework probes four deployment-facing dimensions: procedural reasoning and constraint execution, knowledge grounding and truthfulness, social cognition and affective understanding, and persona fidelity and stylistic consistency. Across more than 35 compressed checkpoints from six open-weight LLMs-covering quantization and pruning-we find that compression redistributes ability profiles. Coarse, local, and surface-level signals often persist, while fine-grained, long-tail, and context-dependent abilities become fragile. Quantization consistently outperforms pruning in retention, but the practical meaning of that gap depends entirely on which ability an application requires. Based on our findings, deployers can choose compression methods by the ability profile their application needs, not just by a leaderboard average. And algorithm designers gain diagnostic signals to build compression methods that preserve what matters. We believe that ACE-Compression turns compression from a black-box accuracy drop into an interpretable intervention-one we can measure, understand, and ultimately improve.

Jiahui Ning, Liang Chen, Yuchen Liang +2

City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1

LLM Compression LLM Evaluation LLM Quantization LLM Pruning

Ask PDF

Jun 1, 2026

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation. However, in safety-critical applications, a model's ability to reliably quantify its own uncertainty is equally important. We ask: does compression preserve this ability? To answer this question, we benchmark 12 LLMs under various compression configurations across five NLP tasks, using conformal prediction to provide a rigorous, distribution-free measure of uncertainty. Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual. These results suggest that accuracy-only evaluation is insufficient for assessing the deployment readiness of compressed LLMs, and that uncertainty-aware benchmarking should be a standard component of model compression pipelines.

Yujia Tong, Yuxi Wang, Yunyang Wan +3

Wuhan University of Technology · Nanyang Technological University

LLM Quantization LLM Evaluation LLM Pruning LLM Compression Confidence Estimation in Language Models LLM Reliability+1

Ask PDF

Jul 5, 2026

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Ask PDF

Abstract

LLM Evaluation

Explore similar work

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations