Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

Aditi Patodiya

cs.SESep 5, 2026

Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

Authors: Aditi Patodiya

Organizations: Independent Researcher

Abstract

Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined documented functional testing of their files. We tested all 305 eligible code-capable GGUF tags (the llama.cpp format) across 15 model lines in the official Ollama library, plus 22 popular Hugging Face community files. A 15-task code screen identified suspects, which we retested on all 164 HumanEval+ tasks. Seven official tags (2.3%) solved at most six of them. All were low-bit quantizations of models with at most four billion parameters. We rebuilt five suspect files from the original weights. All five registry files matched our uncalibrated rebuilds tensor for tensor, so none was corrupted. Exact reproducibility did not imply function. With the weights, toolchain and tensor mix held fixed, adding an importance matrix raised Qwen2.5-Coder-3B scores at three quantization levels from zero to 76-113 of 164 tasks. The same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B functional, showing that the recipe alone does not predict failure. None of 66 sampled tags that passed the screen was non-functional. We release the audit data and the quantcheck tool and recommend screening low-bit tags before publication, then calibrating or withholding failures.

Figures & tables

Work	Object executed	Census	Recipe	Cause
Gao et al. [ 29 ]	API endpoints	–	–	–
Cai et al. [ 30 ]	simulated substitution	–	–	–
Kimi Vendor Verifier [ 31 ]	vendor deployments	–	–	–
Kadasi et al. [ 32 ]	hub task models	–	–	–
Husom et al. [ 34 ]	Ollama tags (edge)	–	–	–
Kurt [ 35 ]	own conversions	–	✓	–

TABLE I: Execution studies. Census : every eligible tag of an official registry. Recipe : controls for the quantization recipe. Cause : controlled attribution of functional failure to file, recipe, model or backend.

Panel of Figure 2 — Fig. 1: Study pipeline. Counts: official library, serial scoring. Only the five Qwen and phi3.5 suspects are rebuilt and replicated on Metal; the two failing community files are tested separately under generic ChatML and the model’s instruction template (RQ2). Unflagged tags are audited on 49 other tasks (RQ4).

Generation	Ollama 0.32.6 [ 56 ] (bundled llama.cpp 96278e3 ), pinned for all generation runs
Rebuilds	llama.cpp v0.7.0 ( 620d222 ) [ 3 ] on CPU: conversion, quantization, imatrix and perplexity
Checkpoints	Qwen2.5-Coder-3B/1.5B/0.5B-Instruct [ 49 ] and Phi-3.5-mini-instruct [ 57 ] at pinned revisions
Backends	NVIDIA RTX 4090 with CUDA (census, reruns); Apple M2 with Metal (replication)
Decoding	greedy, seed 42, context 2,048, at most 768 new tokens
Scoring	EvalPlus 0.3.1, HumanEval+ plus-tests, one evaluation at a time
Calibration	imatrix from 100 chunks of 512 tokens of WikiText-2 train [ 58 ] , or of 1.5 MB of Python source from the llama.cpp repository

TABLE II: Experimental setup.

Class	Count
Healthy (at least 60% of group baseline)	271
Degraded (below 60%, baseline at least 8)	2
Low baseline (baseline below 8 of 15)	6
Suspect (at most 2 of 15, baseline at least 8)	7
No in-frame baseline	19

TABLE III: Classification of the 305 official artifacts against per-variant high-precision baselines (serial scores). All six low-baseline tags, and six of the healthy ones, are deepseek-coder-1.3B tags scoring 1 to 6 of 15.

Fig. 2: 15-task suite score (of 15, serial scoring) for all 305 official-library artifacts, one column per registry model variant and one row per quantization level. Gray: tag not offered at or under 8 GB. Red boxes: the seven suspects. The Qwen2.5-Coder-3B column drops to zero at four adjacent low-bit levels and recovers at 4 bits; deepseek-coder-1.3B scores 1 to 6 at every level, including fp16.

Library tag	Plain	Own imatrix	Third party	Qwen AWQ	Best cal. /164	Verdict
qwen2.5-coder:3b q2_K	0	15, 15	15	14	113	collapse, rescued by calibration
qwen2.5-coder:3b q3_K_S	0	2, 4	4	–	30	collapse, rescue rule not met
qwen2.5-coder:3b q3_K_M	0	9, 8	10	15	125	collapse, rescued by calibration
qwen2.5-coder:3b q3_K_L	0	7, 8	9	–	86	collapse, rescued by calibration
phi3.5:3.8b q2_K	0 (4)	9, 8	7	–	37	collapse, rescued by calibration
llama3.2:1b q2_K	–	–	(1)	–	–	not attributed (no rebuild)

TABLE IV: Attribution of the seven suspects. Scores are 15-task suite passes (of 15) unless marked /164. Plain: our uncalibrated rebuild from the original checkpoint; for phi3.5, with the tag’s tensor mix and, in parentheses, with llama.cpp’s current mix (21 of 164). Own imatrix: our rebuild with an importance matrix from WikiText, from code. Third party: bartowski’s imatrix builds; for llama3.2-1B, unsloth’s files, in parentheses. Best cal.: best calibrated build on all 164 tasks. Our phi3.5 imatrix builds use the current mix and score 9 and 8 against 4 for the plain build at that mix; bartowski’s keeps the tag’s mix and scores 7 against 0.

Fig. 3: Full-benchmark effect of calibration at the levels where the registry tags fail. The gray marker is the registry tag, which equals our uncalibrated rebuild with the registry’s tensor mix (0 of 164 at every level). Colored bars: our calibrated rebuilds and third-party comparisons at the same level. For phi3.5, the hatched bar is our uncalibrated rebuild with llama.cpp’s current mix (21), the like-for-like comparator of our WikiText imatrix build (37); bartowski’s build keeps the registry’s mix (34). Qwen’s AWQ files exist only at q2_K and q3_K_M , no AWQ build exists for phi3.5, and its code-imatrix build was scored on the 15-task suite only (8 of 15). Dashed: the registry’s 4-bit tag of the same model.

Tag	Output
Qwen-3B q2_K	program9 demon1 bil mess(bil is sports Audi program program program… (to the token cap)
Qwen-3B q3_K_S	`python def strlen(s >>> strlen('')` ```
Qwen-3B q3_K_M	. To calculate the length of a given string in Python, . You can use the . Here's a simple function… def strlen(string: str) """Calculate and return the length…
phi3.5 q2_K	import `str`("This is a Python script that solves the following problem in various ways.")))

TABLE V: Openings of four suspects’ answers to HumanEval/23 ( strlen ), verbatim except that runs of whitespace are merged; … marks omitted text.

Level	1.5B	3B
q2_K	13 / 13 / 12	0 / 0 / 15
q3_K_S	15 / 15 / 14	0 / 0 / 2
q3_K_M	15 / 15 / 15	0 / 0 / 9
q3_K_L	15 / 15 / 14	0 / 0 / 7

TABLE VI: The same uncalibrated recipe at two sizes of Qwen2.5-Coder (15-task suite passes): library tag / our plain rebuild / our WikiText imatrix rebuild. Library and plain tensors are identical at both sizes.

Build	Qwen2.5-Coder-3B	phi3.5-mini
fp16	11.5	6.9
Library q4_K_M	12.1	7.2
Library q2_K	66,391	184.1
Library q3_K_S , _M , _L	920, 333, 373	–
Plain rebuild Q2_K , current mix	= library	12.3
Own imatrix Q2_K (W / C)	14.4 / 14.9	10.4 / 10.5

TABLE VII: WikiText-2 test perplexity (20 chunks of 512 tokens; llama.cpp 620d222 on CPU). Library low-bit Qwen files equal our plain rebuilds. W / C: importance matrix from WikiText-2 train / Python code; W is calibrated on the training split of the evaluation corpus; W and C agree within 4%.

Build	Base	Ctx 4k	Ctx 8k	Cap 2k	T0.2
Lib. suspects, plain Qwen-3B	0	0	0	0	0, 0, 0
Plain phi3.5 Q2_K (current mix)	4	4	0	2	3, 4, 3
Lib. Qwen-3B q4_K_M	15	15	15	15	15, 15, 15
Lib. phi3.5 q4_K_M	11	11	11	9	11, 11, 11

TABLE VIII: 15-task suite passes (of 15) under alternative generation settings. Base: greedy, context 2,048, 768-token cap. T0.2: temperature 0.2 with seeds 1, 2 and 3. Lib.: registry library tag; suspects: the five rebuilt ones. Plain Qwen-3B: our Q2_K and Q3_K_M rebuilds.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 802 papers

What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains
Oct 14, 2026Wei-Liang Liu, Cheng-Yen TsaiLLM Security ML Reproducibility
National Yang Ming Chiao Tung University · National Taiwan University of Science and Technology
IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation
Feb 26, 2026Yanpei Guo, Wenjie Qu, Linyu Wu +7Verifiable Outsourced Computation LLM Auditing
National University of Singapore · Nanyang Technological University · Independent Researcher +1
Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit
Sep 25, 2026Jakub PrejznerLLM Tool Use LLM Quantization
Independent Researcher

Oct 14, 2026J/K move · Enter open · S save

What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains

Wei-Liang Liu, Cheng-Yen Tsai

National Yang Ming Chiao Tung University · National Taiwan University of Science and Technology

LLM Security ML Reproducibility LLM Quantization

Open paper AskPDF arXiv

Model signing authenticates the full-precision weights W an LLM is released with, yet users run a quantized artifact Q whose producing transformation is not uniquely determined under current disclosure. We separate the non-environment inputs from the execution environment E and measure a quantity that, to our knowledge, has not been measured before: the Independent Verifiability Rate (IVR), the fraction of a popularity-sampled GGUF ecosystem that a third party can re-derive bit-for-bit. Across the 200 most-downloaded Hugging Face GGUF repositories (198 in scope), no artifact or model card records the build environment, so under a recorded-inputs protocol the IVR is 0/198. Yet the artifacts are not irreproducible. On four in-sample repositories that disclose their inputs, the card's information alone reproduces no Q4_K_M file, but two coordinates the card omits - the build's floating-point-contraction setting and, for hybrid models, the per-tensor quantization recipe - re-derive 7/8 target-quant files bit-for-bit on x86 (5/8 on arm64); the fp16 re-derives from the provider's checkpoint, but the calibration imatrix must be shipped and hash-bound, since recomputing it did not match. An older release additionally pins the instruction set and did not re-derive. Reproducing the divergence reported with llama.cpp PR #25353, we show that E is a necessary coordinate and that one compiler setting yields a unique output across the tested platforms at negligible quality cost (+0.06% perplexity). As a security characterization, benign floating-point non-determinism separates from uniformly placed nibble-flip tampering above a detection floor that grows with the environments a verifier must allow, but a targeted embedding edit evades the statistic, so the proposed protocol rests on exact re-derivation; adversarial-fp16 attacks lie outside it. We release the audit data and a re-runnable harness.

Abstract

LLM Reliability

Explore similar work

What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains

IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit