Post-training quantization to the GGUF format's mixed-precision K-quants is commonly how open-weight language models reach consumer hardware, yet its effect on fine-grained lexical competence is uncharacterized. We audit 27 quantized artifacts across 13 families and four architecture backbones, 0.35B-14B parameters, evaluated down their published ladder to Q2_K (about 2.6 bits per weight), on 429 frequency-validated rare English words under two probes: surface inclusion of a prompt-supplied word and its one-sentence definition, scored by a tiered multi-synonym matcher, its error measured by a blind LLM-judge census of every definition, with human verification. Three regimes emerge at Q2: total collapse into unusable builds, severe semantic dissociation in sub-2B models, and mostly robust preservation above about 3B. In every sub-2B artifact, definitions fall 20-67% below the artifact's baseline, typically several times the inclusion loss. Two controls separate rarity from task difficulty: within the rare set, loss rises with rarity in six of seven sub-2B artifacts, and on a 100-word common-word set rare words lose more than common words in all eight, significantly in six. Tokenizer vocabulary size does not predict the damage (Spearman rho=0.12); parameter count dominates (rho=0.72), confirmed within five of six same-tokenizer families. Q4_K_M remains lexically clean at >=1B. The damage is frequency-graded, provider-dependent, and not calibrated by WikiText-2 perplexity: across nine artifact-matched ladders, near-identical Q2 penalties (44.7%/47.6%) separate an artifact keeping its definitions (3.6%) from one losing them (43.6%). Aggressively quantized small models can keep generating fluent text while no longer knowing what it means, risking hardware-constrained deployments in domains where semantics carries consequences. Validation must be per artifact.
Figures & tables
Figure 1: Silent semantic failure, verbatim from the released outputs. Granite-4.0-H-1B (Table 1 ) on the rare word dirge , FP16 baseline vs. its Q2_K build. At FP16 the definition is correct; at Q2_K the model reports the word as unrecognized, yet still produces it fluently in every Probe A sentence (surface inclusion =1.00 at Q2). In another run it is a medicine (“The doctor prescribed a dirge for her patient’s health issues”). Both census judges independently mark the Q2_K definition wrong. Quotes are verbatim; the word and full outputs are in the released definitions.csv and generations.csv for Experiment 34
Figure 2: Study design and scale. Left to right: the stimulus sets (429 Zipf-validated rare words; a 100-word common-word control); the 27 instruction-tuned quantized artifacts (13 model families, four architecture backbones), each evaluated across its published GGUF ladder (five to eleven levels per artifact), built by two providers (13 Ollama-native, 14 Unsloth via Hugging Face); with the build-source and common-word controls, 257 file-level evaluations in total (221 rare-word, 36 common-word); the two word-paired probes; the tiered scorer (t0/t1/t2), validated by a two-judge LLM census of every definition and human spot-checked; and the three-regime outcome at Q2. In total the sweep produced ≈ 98,400 word-level definition judgements and ≈ 295,200 inclusion generations ( ≈ 393,600 scored outputs), all on a single GPU within a 12–16 GB VRAM budget (RTX 3060, 12 GB; Kaggle T4, 16 GB)
Figure 3: Relative drop at Q2 on both probes, for the 26 primary artifacts with a published Q2 build ( ‡ ). (a) Probe A (surface inclusion, x ) against Probe B (definition, y ). Regime-II artifacts hug the y -axis except the three IIb members (Llama-3.2-1B, Qwen2.5-1.5B, Phi-4-mini-3.8B), whose larger inclusion loss shifts them rightward; the collapse cohort sits on the diagonal near 100%; robust models cluster at the origin. The Qwen2.5-0.5B point at negative x is the repetition effect of Table 1 . (b) The fifteen artifacts inside that origin cluster, one row each
Figure 4: (a,b) Accuracy across the quantization ladder for one exemplar of each regime (K_M files). Surface inclusion is flat through Q3 for all three; only the definitional probe separates the regimes. (c) For the selective exemplar (Granite-4.0-H-1B), WikiText-2 perplexity (diamonds) degrades smoothly while the definitional probe (squares) declines discontinuously at Q2 and inclusion (circles) barely moves
Figure 5: Accuracy change against baseline at every evaluated precision level, both probes, showing the worst sub-variant per level, models ordered by parameter count (“–” marks a level not evaluated). (a) Probe B (definitional, Tier-2) for all 27 artifacts, 216 files; asterisks mark raw per-comparison McNemar p<.05 , and 30 of the 40 starred cells survive study-wide Benjamini–Hochberg correction. (b) Probe A (surface inclusion); asterisks mark word-level Wilcoxon p<.05 . Severe inclusion losses occur only in the three Regime-I builds and the three Regime-IIb artifacts; the +17 cell is the Qwen2.5-0.5B repetition effect of Table 1
Figure 6: Absolute probe trajectories across the quantization ladder for all 27 artifacts, ordered by parameter count. Probe B (definitional accuracy, Tier-2) is a solid line with circle markers; Probe A (surface inclusion) is a dashed line with square markers. On the Probe B line, filled circles are the standard K_M file at each level and open markers the K_S ( ▽ ) and K_L ( △ ) sub-variants where published. Colour gives the Q2 regime of Table 1 ; Granite-4.0-H-350M publishes no Q2 build and ends at Q3. Qwen2.5-0.5B and Qwen2.5-1.5B are the only artifacts whose Probe A baseline is below their Probe B baseline, causing their lines to cross. The rise in Probe A for Qwen2.5-0.5B at Q2 reflects the repetition effect reported in Table 1
Figure 7: Common-word control: relative change in definitional accuracy (Probe B, Tier-2) at Q2 ( ‡ ) for eleven artifacts, solid bars for the 429 rare words and striped bars for the 100 common words. Word-type × quantization interactions are given in the text
Figure 8: Frequency dose-response within the rare-word set. Tracks words defined correctly at the baseline to show the probability of definitional loss at Q2, grouped by Zipf-frequency tercile. Line colour = architecture class, marker = model. Loss probability rises with rarity in all six (logistic Zipf coefficients −0.51 to −1.10 , all p≤.013 ). Qwen2.5-0.5B (not shown) is frequency-flat
Figure 9: Predictors of definitional fragility across the 23 non-collapse primary artifacts; colour = architecture class. Left: Probe B relative change at Q2 against parameter count (log scale); Spearman ρ=0.72 , p=1×10−4 . Right: the same quantity against tokenizer vocabulary size; ρ=0.12 , p=.59 . Parameter count is associated with damage; vocabulary size is not
Figure 10: Same-tokenizer scale ladder across the Qwen2.5 family (0.5B, 1.5B, 7B; identical 152k vocabulary), both probes at Q2. (a) Relative change. (b) The same ladder on the odds scale, which is insensitive to baseline differences: the 0.5B departure is not an effect of that artifact’s low baseline, since its gap to 1.5B widens from ×3.3 to ×9.3 . The 3B point (open cross, panel a) is the collapsed Q2 build, excluded from the trend; the positive 0.5B inclusion value is the repetition effect of Table 1
Figure 11: Within-family scale ladders: Probe B relative change against parameter count, one line per family (colour and marker shape both distinguish family), each compared at the most aggressive precision level all of that family’s members were evaluated at (given in the legend; Granite-4.0-H at Q3 ‡ ). Qwen2.5’s collapsed 3B build is omitted from this panel; it appears as the dotted trace in Fig. 12
Figure 12: The same six families across every evaluated precision level, one panel per family and one line per member, with darker shading for larger parameter counts. Each point is the worst sub-variant evaluated at that level. Where the larger member’s trajectory falls below the smaller member’s it does so by under 3.5 percentage points, except in Qwen2.5 at Q2. The dotted Qwen2.5 trace (3B ∗ ) is the collapsed build
Figure 13: Replication of the perplexity comparison of Fig. 4 c in the two other dissociating sub-2B ladders: (a) LFM2.5-1.2B, (b) Qwen2.5-1.5B (K_M files); Perplexity (diamonds) rises along the familiar accelerating quantization curve while the definitional probe (squares) drops sharply at Q2; surface inclusion (circles) moves far less
Figure 14: The full perplexity program: nine artifact-matched WikiText-2 ladders (K_M files; Granite-4.0-H-7B from its Q8 baseline). (a) Q2 endpoints, perplexity change (log scale) vs. relative definitional change. Nearly matched penalties (shaded band) separate the robust Qwen2.5-7B from the dissociating Granite-4.0-H-1B; (b) Granite-4.0-H-1B (filled, solid) vs. Qwen2.5-7B (open, dashed) on shared levels: near-coincident perplexity curves, opposite definitional outcomes
Figure 15: Artifact-source sensitivity: relative change at Q2 on both probes for Llama-3.2-1B in two independently quantized GGUF builds of the same model (Ollama vs. Unsloth). The definitional loss’s direction replicates, while magnitude differs across builds