cs.AIOct 19, 2026

When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders

Authors: Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon

Organizations: ThakiCloud · Korea University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learned post-training quantization (PTQ) calibration fits quantizer parameters on a small calibration set, and for a text embedder the natural supervision is retrieval relevance. We ask two questions that are usually conflated: how far to calibrate, and whether to calibrate at all. Across three embedders (e5-base-v2, BGE-M3, EmbeddingGemma-300m) at 4-bit weight quantization, 161 calibration runs, the outcome of a fixed procedure ranges from a gain to catastrophic degradation, and the driver is the dose of the update, measured as the functional displacement D_func of the quantized model from its uncalibrated start. On EmbeddingGemma the same learning-rate grid gives +1.47 to -71.53 points of in-domain NDCG@10 change; a label-free cosine objective collapses at the same dose (-69.52 vs -71.63), so the collapse is a property of learnable calibration rather than of retrieval supervision. Objective sensitivity is heterogeneous: on e5, ranking calibration is substantially more robust than cosine at matched displacement (+14.16 +/- 4.34 pp, 95% t-CI [+3.37, +24.96], three paired slates), whereas on BGE-M3 the ordering is directionally opposite but unresolved ([-5.18, +2.38]). Dose is the main axis but not a sufficient statistic: equal D_func reached along different paths differs by up to 1.61 pp. We therefore treat calibration as an optional intervention. A pre-registered rule that accepts a checkpoint only when a per-query paired confidence interval on a 25% calibration-validation split clears the uncalibrated baseline was run on 6 independent slates and five arms. It never selected a checkpoint below its slate's baseline (0 harmful selections in 30 decisions), retained 8 of 11 beneficial updates, and paid a mean oracle regret of 1.52 pp with one abstention that forfeited 11.13 pp, a safety-opportunity trade-off, not a free improvement. We report what remains open, including label efficiency.

Figures & tables

Figure 1: In-domain disjoint NDCG@10 change (pp, final step vs step 0) against functional displacement DfuncD_{\mathrm{func}} at the final step, one point per learning rate (median of three optimizer seeds), LambdaRank objective. (a) the small-dose region where every model gains; (b) the full range. DfuncD_{\mathrm{func}} is not comparable across models—each model is its own series—but within a model the damage is monotone in dose. EmbeddingGemma reaches Dfunc≈1D_{\mathrm{func}}\approx 1 at 3×10−33{\times}10^{-3} and loses −- 71.53  pp\,\mathrm{pp} .
ModelDfuncD_{\mathrm{func}} binrankingcosinegap
Gemma0.00–0.05++ 0.88++ 1.19++ 0.31
 0.05–0.10++ 1.27++ 1.08−- 0.19
 0.10–0.20++ 0.09−- 3.97−- 4.07
 0.20–0.35−- 3.53−- 11.77−- 8.24
 0.35–0.50−- 12.18−- 13.55−- 1.37
 0.50–0.75−- 26.09−- 23.62++ 2.48
Table 1: Pre-registered dose-matched comparison of the ranking objective against the label-free cosine objective, per model and DfuncD_{\mathrm{func}} bin: mean in-domain disjoint change (pp) of all trajectory points falling in the bin, and the gap (ranking −- cosine). Tolerance is three times each model’s own σopt\sigma_{\text{opt}} ; the verdict is the comparator’s, computed on slate 0 trajectories. EmbeddingGemma bins 0.20–0.75 come from the added intermediate learning rates. Verdicts: Gemma not_retrieval_specific (7 bins, tolerance 1.51  pp\,\mathrm{pp} ), e5 control_worse (3 bins, 0.58  pp\,\mathrm{pp} ), BGE-M3 retrieval_specific (4 bins, 0.92  pp\,\mathrm{pp} ).
ModelcellsΔ\Delta per slatemean ±\pm SD95% t-CIinterp.max ∣ΔD∣|\Delta D|
e5rank 3e-3 / cos 1e-2 ( D≈0.15D{\approx}0.15 )++ 16.98 / ++ 9.16 / ++ 16.35++ 14.16 ±\pm 4.34[ ++ 3.37, ++ 24.96]++ 14.080.016
BGE-M3rank 1e-3 / cos 3e-3 ( D≈0.21D{\approx}0.21 )−- 3.14 / −- 0.70 / −- 0.35−- 1.40 ±\pm 1.52[ −- 5.18, ++ 2.38]−- 1.530.008
Table 2: Paired-slate, step-matched objective comparison. For each of three independent calibration slates the two decisive cells per model are run; on the higher-dose arm the measured step whose DfuncD_{\mathrm{func}} is nearest the lower arm’s final DfuncD_{\mathrm{func}} is selected (caliper 0.02, met by every pair), and Δ\Delta is the ranking-minus-cosine in-domain disjoint change at that matched dose. ±\pm is the SD over slates; the interval is a 95% tt -CI with n=3n{=}3 . “interp.” repeats the analysis with linear interpolation between adjacent measured steps and is a sensitivity check, not an observation.
Figure 2: Validation-gated selection on six independent calibration slates. For each arm (model, learning rate) the filled series is the in-domain disjoint change at the final step without a gate and the open series is the change at the checkpoint the rule selected (0 when it abstained); error bars are SD over the six slates. The gated series never goes below zero; the price is visible on the e5 arms, where the gate sometimes abstains from a large gain.
Modellrungated finalgated Δ\Deltamin gatedabstainregret
EmbeddingGemma1e-4++ 1.5 ±\pm 1.4++ 0.78 ±\pm 0.90++ 0.003/61.12
EmbeddingGemma3e-4−- 11.2 ±\pm 2.2++ 0.66 ±\pm 1.05++ 0.004/61.33
EmbeddingGemma1e-3−- 31.9 ±\pm 3.5++ 0.18 ±\pm 0.43++ 0.005/60.63
e5-base-v21e-3++ 6.6 ±\pm 1.7++ 5.53 ±\pm 2.92++ 0.001/62.33
e5-base-v23e-3−- 7.0 ±\pm 4.5++ 5.29 ±\pm 2.73++ 0.001/62.17
all 30 decisions: harmful selections 0 ⋅\cdot abstained 14 ⋅\cdot opportunity 8/11
Table 3: Six-slate scorecard, five arms. “ungated final” is the change at step 12 without a gate, “gated Δ\Delta ” the change at the selected checkpoint (abstention scores 0), both mean ±\pm SD over six slates; “min gated” is the worst selected checkpoint on any slate; “regret” is the mean oracle regret over all six decisions of the arm, with abstention scored at the baseline. The pre-registered harm-prevention score counts only abstention on arms whose final step is catastrophic and is kept unchanged beside the realized outcome.
calibration setnnheld-out NDCG@10labels read
128 random30.4755 ±\pm 0.0350128
128 selected30.4835 ±\pm 0.01431,024
512 random30.4831 ±\pm 0.0183512
512 selected30.4804 ±\pm 0.02001,024
128 sel. −- 512 rnd.: ++ 0.0004 NDCG
Welch 95% CI [ −- 0.037, ++ 0.038]
Table 4: Label-budget comparison (PRD gate “128 selected ≈\approx 512 random, difference ≤0.2\leq 0.2 NDCG”), in the gate’s own unit. Held-out NDCG@10 is the mean over ArguAna and SciDocs; “labels read” is the number of relevance judgements the run accessed, including the candidate pool screened by the selection heuristic.
More in this topic

Post-Training Quantization

Post-Training Quantization compresses trained neural networks into low-bit weights, activations, or caches to reduce memory use and inference cost.

See all 360 papers

Explore similar work

CardsList
Sep 14, 2026

Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families

Weight-only post-training quantization is the cheapest way to shrink a retrieval embedder, and the received advice for applying it -- protect the embedding table, allocate bits by module sensitivity, prefer a ranking-aware objective over weight reconstruction -- was carried into LLM quantization largely intact. We test that advice on retrieval embedders directly, quantizing five checkpoints from four architecture families across a grid of bit widths and group sizes, and isolating the embedding, attention and feed-forward blocks at each width. Every heuristic fails to transfer as stated. The embedding table never emerges as the dominant isolated protection priority in any family, despite being the largest tensor in several of them. Module sensitivity does not survive as a transferable ordering: at INT4/g16 the spread between modules is too small to allocate against, at INT3 the ordering becomes family-dependent and joint damage stops being the sum of its parts, and at INT2 comparable reconstruction error accompanies retention ranging from 1.3 to 65.9 percent of full precision. A cheap reconstruction proxy is useful for screening uniform bit widths but substantially less reliable for choosing which tensors to protect; its apparent strength across the whole grid is a range-extension artifact. A distilled 109M student at INT3 holds 78.04 NDCG@10 in 68.4 MB and dominates the extreme-PTQ arm of its own 0.6B teacher, 297.9 MB at 64.46, on both size and quality -- but only inside the task it was distilled for. Sizes are byte counts of files that exist rather than arithmetic estimates, and the measurement repository carries the byte provenance for every one of them.

Hyojung Han

ThakiCloud

Text Embedding EvaluationWeight-Only QuantizationText Embedding ModelsPost-Training QuantizationNeural Network Quantization
AskPDF
Apr 27, 2026

Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels

Post-Training Quantization (PTQ) compresses large language models to low bit-widths using a small calibration set, and its quality depends strongly on which samples are chosen. We identify a failure mode in which calibration samples fail to activate outlier channels, hidden dimensions with unusually large activations, causing the quantizer to underestimate their dynamic range and producing per-channel reconstruction errors that dominate layer-wise loss. Motivated by this observation, we argue that PTQ calibration quality is governed more by weighted outlier-channel coverage than by generic sample representativeness, and formulate calibration selection as a weighted set cover problem over outlier channels. The objective is monotone submodular, and the greedy algorithm, COVERCAL, operates on pre-computed activation statistics and requires no GPU time at selection. We further show that the weight choice is internally consistent: under a stylized clipping model, missed weighted coverage upper-bounds surrogate loss, justifying the weighted coverage objective as principled rather than purely empirical. Across LLaMA-2, LLaMA-3, and Mistral, under AWQ and GPTQ backends and five downstream evaluations, COVERCAL improves over random, max-perplexity, max-activation-variance, and stratified baselines, with the largest gains at small calibration budgets. At INT4 with 128 samples, COVERCAL improves MMLU by 1.2 to 1.5 points over random calibration and reduces perplexity degradation by 15 to 30%; with 64 samples, it matches or exceeds random calibration at 256. The contribution is not a new PTQ backend but a formulation of calibration selection as weighted outlier coverage, with a simple, efficient algorithm and a surrogate-based justification.

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

LLM QuantizationActivation OutliersPost-Training Quantization
AskPDF
Oct 6, 2026

Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders

Mixed-precision post-training quantization needs a per-module sensitivity signal; for a text embedder the obvious one -- the retrieval quality a module costs when quantized -- needs relevance labels that deployments rarely have. We measure a label-free substitute: quantization-induced representation drift, obtained by quantizing one module, re-encoding the corpus, and recording how far the output embeddings moved from their full-precision positions. What is specific is the observable: the deployed output representation a dense retriever ranks with. Across five development embedders, configuration-level drift orders sampled mixed-precision plans against held-out retrieval quality at a macro Spearman of 0.911, the sensitivity transports across calibration corpora and retrieval domains in the usable regime, module drifts compose rank-consistently but not numerically, and relevance-derived sensitivity adds no consistent value. The method is one additive allocation under a hard packed-byte budget, with no labels and no search. On three embedders held untouched until method, baselines and hypotheses were frozen and sealed, the pre-registered directional hypothesis against the prior LieQ criterion holds (3/3 at the main budget, no collapse) and drift scores above a two-sided LieQ steelman in 2/3; but at the main budget drift is numerically lower than same-budget uniform precision on all three (-0.99, -0.85, -1.01 points), having reduced module and whole-model drift as designed. Output drift is thus a robust coarse sensitivity signal, not a universally optimal allocation objective: it avoids the catastrophic failures of the transferred signed-geometry adaptation and can remain usable at stressed budgets where uniform collapses, but fine-grained redistribution around a strong uniform operating point remains unresolved.

Hyojung Han, Jongmin Kim, Seung-Hun Jeon

ThakiCloud

Post-Training QuantizationMixed-Precision QuantizationNeural Network QuantizationRepresentation Drift
AskPDF