ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

Jason Z Wang

cs.LGApr 15, 2026

ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

Authors: Jason Z Wang

Organizations: Independent

Abstract

At matched accuracy, open-weight LLMs differ substantially in the shape of their error severity distribution -- a difference invisible to the scalar error rate. Hallucination benchmarks report a single error count and treat all errors as equivalent, yet a wrong date and a fabricated court ruling differ by orders of magnitude. We introduce Errorquake-10k, a 10,000-query benchmark scoring each response on a continuous 0-4 severity scale across 8 domains and 5 difficulty tiers, and we fit per-model severity distributions for 21 open-weight models. For each model we estimate a severity distribution index (b, the Gutenberg-Richter upper-tail slope) with 95% bootstrap confidence intervals. Headline: across the 210 model pairs, 85 have disjoint 95% b confidence intervals at matched accuracy (|Delta epsilon| < 0.05) on human-consensus scoring, e.g. deepseek-v3.2 vs. ministral-14b at epsilon = 0.586 and Delta b = 0.47. A 519-item three-rater human validation study confirms measurement reliability (ICC(2,k=3) = 0.85), validates the LLM-judge ranking (rho = 0.89), and confirms the dense-model scaling correlation on human data (rho_s = -0.86). We prove a Non-Reducibility Theorem showing that severity profile and error rate are informationally non-redundant (I(b; model | epsilon) = 1.56 bits; 64.5% of cross-model b variance is unexplained by epsilon). A severity mechanism taxonomy (kappa = 0.83) reveals that error type shifts categorically with severity: low-severity errors are retrievals (71%); high-severity errors are fabrications (39%) -- and this composition differs by model size (p < 0.0001). Severity distribution should be reported alongside accuracy; it carries discriminative information that the error rate cannot.

Figures & tables

Experiment	Criterion	Observed	Verdict
Exp. 1 (distributions)	non-exp. best fit or Vuong-decisive $\geq 70\%$	$17/21$ non-exp; $17/21$ Vuong-dec.	PASS ‡
Exp. 2 (discriminator)	$\geq 3$ disjoint-CI pairs	$85$ pairs (human) / $31$ (judge)	PASS
Exp. 3 primary	$\rho_{s}\geq 0.75$ at $M\geq 3$	$\rho_{s}=0.443$ , $p{=}0.044$	FAIL
Exp. 3 secondary	within $1.5\times$ for $\geq 65\%$	$19\%$	FAIL
Exp. 5 (scale null)	no clean correlation	$\rho_{s}=-0.562$ , $p{=}0.006$	REJECTED
Sensitivity S1	ranking $\rho>0.85$ under coarsening	$0.43$ / $0.16$	FAIL

Table 1: Pre-registered criteria and outcomes. Exp. 5’s null was rejected in the opposite direction from intuition. † S2 misses the threshold by

0.003

and is reported as “borderline fail”. ‡ Four models are BIC-best-fit by exponential, but

17/21

are non-exponential and

17/21

are Vuong-decisive; taken together, every model shows tail-shape structure under at least one criterion.

Figure 1: Magnitude-frequency curves for four representative models spanning the

b

range (heaviest to lightest tail). Dashed red: Gutenberg–Richter fit. All

21

models and the

\Delta

BIC heatmap are in Appendix C .

Figure 2:

b

vs. active parameter count. Dense (blue) shows a monotone downward trend. Reported as a sensitivity observation; the headline is § 4.1 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

metric	4K (v5/v6)	10K (v7)
Total queries per model	4000	10000
Disjoint-CI matched-accuracy pairs (Exp. 2)	30	31
Qualifying pairs $\|\Delta\varepsilon\|<0.05$ , $\|\Delta b\|>0.15$	42	44
Models with valid $b$ fit	21	21
Vuong-decisive distribution best-fit	18	17
Non-exponential best-fit	17	17

Appendix

Table 2: 4K-vs-10K comparison of the LLM-judge baseline metrics.

Figure 3: Predicted vs. observed catastrophic counts per model for Experiment 3. Left: pre-registered

M\geq 3.0

threshold,

\rho_{s}=0.443

. Right: exploratory

M\geq 2.5

threshold,

\rho_{s}=0.637

. Solid line is identity, dashed lines mark

\pm 1.5\times

. Only

4/21

models land within

1.5\times

M\geq 3.0

;

0/21

over-predict.

Figure 4: All

21

models sorted by

b

(heaviest-tailed at top). Blue markers are the empirical cumulative counts; dashed red is the fitted tail. Compact form of Figure 1 .

Figure 5: $\Delta$ BIC between each distribution family and the best-fit family for each of the $21$ models, capped at $60$ . Stars mark the selected family.

Figure 6: $b$ by model $\times$ domain. Red = heavier tail. Per-model rows are sorted by mean $b$ . Kendall $W=0.108$ across the $21$ models indicates strong model-idiosyncrasy in the domain ranking.

model	$b$	$95\%$ CI	$m_{\min}$	$n_{\geq m_{\min}}$	$n_{\text{err}}$	$\varepsilon$	best fit
deepseek-v3.2	0.595	[0.582, 0.609]	0.5	5548	5548	0.580	stretched exp
gemma-2-27b	0.631	[0.617, 0.644]	0.5	6307	6307	0.655	stretched exp
gpt-oss-20b	0.887	[0.796, 1.000]	2.5	171	5865	0.607	trunc. power law
mistral-small-4-119b	0.900	[0.796, 1.019]	2.5	142	5831	0.603	exponential
seed-oss-36b	0.904	[0.811, 1.023]	2.5	126	5808	0.602	stretched exp
gemma-3-12b	0.917	[0.843, 1.005]	2.5	266	6257	0.650	stretched exp

Appendix

Table 3: Full 21-model

b

table. Bootstrap CIs from

n=2000

resamples.

model dropped	LOO Spearman $\rho$	$p$ -value
ministral-14b	$-0.777$	$0.0018$
gemma-3-4b	$-0.736$	$0.0042$
solar-10.7b	$-0.725$	$0.0051$
gemma-3-12b	$-0.711$	$0.0065$
llama-3.2-3b	$-0.705$	$0.0071$
mistral-small-24b	$-0.704$	$0.0072$

Appendix

Table 4: Leave-one-out Spearman correlations on the Exp. 5 headline. All

14

drops preserve sign and significance. Baseline

\rho_{s}=-0.562

p=0.006

(

n=14

pair	hypothesis	$\Delta b$	$95\%$ CI	$p$	verdict
llama-3.2-3b vs 3.1-8b	within-Llama scale	$-0.532$	$[-0.749,\,-0.287]$	$0.000$	sig
qwen2.5-7b vs llama-3.1-8b	family at $8$ B	$-0.317$	$[-0.556,\,-0.040]$	$0.021$	sig
gemma-3-12b vs 27b	within-Gemma-3 scale	$-0.198$	$[-0.389,\,+0.008]$	$0.057$	marg.
gemma-3-12b vs ministral-14b	family at $\sim 13$ B	$-0.189$	$[-0.408,\,+0.030]$	$0.091$	n.s.
gemma-3-4b vs 12b	within-Gemma-3 scale	$-0.169$	$[-0.453,\,+0.069]$	$0.205$	n.s.
mistral-24b vs gemma-3-27b	family at $\sim 25$ B	$-0.132$	$[-0.378,\,+0.127]$	$0.301$	n.s.

Appendix

Table 5: Training-pipeline pair tests.

2/11

pairs reach

p<0.05

; notably, the DeepSeek version bump v3.1

\to

v3.2 does not significantly change the severity distribution.

Figure 7: Gemma-2 vs. Gemma-3 at 27B (generation upgrade). Within-family generation comparison; the

\Delta b

is small and not statistically significant (Table 5 ).

Figure 8: DeepSeek v3.1 vs. v3.2 (minor version bump). The two versions are statistically indistinguishable in tail shape (

p=0.872

, Table 5 ).

$T$	$n_{\text{surv}}$	$n_{\text{dense}}$	discrim. pairs	$\rho_{s}^{\text{scale}}$	$p$	partial $\rho_{s}{\mid}\varepsilon$	partial $p$
$30$	$21$	$14$	$30$	$-0.562$	$0.006$	$-0.270$	$0.35$
$50$	$18$	$13$	$22$	$-0.704$	$0.007$	$-0.207$	$0.32$
$75$	$11$	$8$	$13$	$-0.850$	$0.008$	$-0.548$	$0.16$
$100$	$11$	$8$	$13$	$-0.850$	$0.008$	$-0.548$	$0.16$
$150$	$9$	$6$	$10$	$-0.928$	$0.008$	$-0.600$	$0.21$
$200$	$9$	$6$	$10$	$-0.928$	$0.008$	$-0.600$	$0.21$

Appendix

Table 6: Exceedance threshold sweep. At every tested threshold the Exp. 2 discriminator count exceeds the pre-registered criterion of

\geq 3

disjoint-CI pairs (minimum

10

T=200

). The dense scaling correlation strengthens as the threshold tightens (from

-0.562

-0.928

) because the survivors are the best-supported tail fits; the partial correlation controlling for

\varepsilon

weakens with the shrinking sample and does not reach

p<0.05

at any threshold, consistent with the underpowered regime.

model	$\varepsilon$	$b$	$M{\geq}1.5$	$M{\geq}2.0$	$M{\geq}2.5$	$M{\geq}3.0$	$M{\geq}3.5$	$M{\geq}4.0$
qwen3-next-80b	0.592	1.052	122,061	56,278	10,755	3,252	250	0
llama-3.2-3b	0.450	1.046	315,695	275,366	81,428	52,915	13,357	4,110
gpt-oss-20b	0.586	0.938	160,991	78,368	17,026	5,759	1,252	250
phi-3.5-mini	0.285	1.309	202,076	197,012	17,220	13,928	2,279	0
gemma-3-4b	0.387	0.979	334,883	305,964	101,730	62,742	22,205	2,066
qwen2.5-7b	0.316	1.257	214,828	211,285	21,761	17,206	3,289	0

Appendix

Table 7: Expected event counts per

1{,}000{,}000

queries at six severity thresholds, computed from the

10{,}000

-query empirical evaluation. Wilson

95\%

binomial CIs and raw counts are in results/analysis/deployment_table.json . Rows sorted by

\log_{10}

(active params). No fit, no extrapolation.

Figure 9: Empirical event rate per million queries for

21

models, sorted by catastrophic count (

M\geq 3.0

, red bars) with severe (

M\geq 2.5

, orange bars) for comparison.

tier	$n$	dense $\rho_{s}(\log_{10}\text{params},b_{\text{tier}})$	$p$
T1 (easy)	14	$-0.493$	$0.073$
T2 (easy)	14	$-0.562$	$0.037^{*}$
T3 (intermediate)	14	$-0.322$	$0.262$
T4 (hard)	14	$+0.046$	$0.875$
T5 (hard)	14	$-0.425$	$0.130$
aggregated (all 5)	14	$\mathbf{-0.562}$	$\mathbf{0.006}$

Appendix

Table 8: Per-tier scaling correlation on the

14

dense models. Sign preserved in

4/5

tiers; T2 individually significant. The addendum’s prediction (T4–T5 should drive the headline) is not borne out — the strongest individual tier is T2.

model	$b$ ${}_{\text{def}}$	$b$ m=1.5	$b$ m=0.5	$n_{\geq 0.5}$
llama-3.2-3b	1.046	0.469	0.290	1750
phi-3.5-mini	1.309	0.530	0.298	1126
gemma-3-4b	0.979	0.439	0.243	1497
qwen2.5-7b	1.257	0.517	0.299	1247
llama-3.1-8b	1.001	0.737	0.566	2429
eurollm-9b	1.067	0.526	0.297	1356

Appendix

Table 9: S5: dense

b

under three

m_{\min}

strategies. The default model-specific KS-selector targets the upper tail; fixed

m_{\min}=1.5

and the Clauset-style

\geq 100

-exceedances rule both target the bulk of the positive distribution. The sign of the scaling correlation flips between the two regimes: larger dense models have heavier upper tails but lighter bulk decay rates.

Figure 10: S1:

b

under scale coarsening. The 9-point grid is load-bearing; both coarsenings fall well below the

\rho>0.85

stability threshold.

Figure 11: S2: ranking stability under overcall correction. Mean Spearman

\rho=0.847

across

50

bootstrap trials, narrowly below the pre-registered

0.85

threshold.

Figure 12: Per-model overcall breakdown on the

340

-item human-rated subset (Appendix L ).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms
Mar 9, 2026JV RoigHallucination in Language Models LLM Evaluation
Kamiwaza AI
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM Evaluation Language Model Generation Evaluation
1Martian · 3ThoughtWorks · University of Oxford

Apr 28, 2026J/K move · Enter open · S save

The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive

Alex Bogdan, Adrian de Valois-Franklin

Evolutionairy AI Toronto, Canada

LLM Evaluation Language Model Generation Evaluation LLM Auditing Language Model Fingerprinting Anomaly Detection

Open paper AskPDF arXiv

We report a striking statistical regularity in frontier LLM outputs that enables a CPU-only scoring primitive running at 2.6 microseconds per token, with estimated latency up to 100,000

\times

(five orders of magnitude) below existing sampling-based detectors. Across six contemporary models from five independent vendors, two generation sizes, and five held-out domains, token rank-frequency distributions converge to the same two-parameter Mandelbrot ranking distribution, with 34 of 36 model-by-domain fits exceeding

R^{2} = 0.94

and 35 of 36 favoring Mandelbrot over Zipf by AIC. The shared family does not collapse the models into statistical duplicates. Fitted Mandelbrot parameters remain cleanly separable between models: the cross-model spread in

q

(1.63 to 3.69) exceeds its per-model bootstrap standard deviation (0.03 to 0.10) by more than an order of magnitude, yielding tens of standard deviations of separation per few thousand output tokens. Two capabilities follow. First, statistical model fingerprinting: text from a vendor-delivered LLM can be tested against its claimed model family without cryptographic watermarks or access to model internals, supporting provenance verification and silent-substitution audits. Second, a model-agnostic reference distribution for black-box output assessment, from which we derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs. Pilot results on FRANK, TruthfulQA, and HaluEval map where the primitive helps (lexical anomalies, unsupported entities) and where it structurally cannot (reasoning errors in domain-appropriate vocabulary). We position the primitive as a first-pass triage layer in compound evaluation stacks, not as a replacement for sampling-based or source-conditioned verifiers.

Abstract

LLM Evaluation

Explore similar work

The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

The Capability Frontier: Benchmarks Miss 82% of Model Performance