How Far Does Second-Order Linear Response Predict Quantization Mismatch? Pre-Registered Measurements on Five Language Models
Organizations: University of Tübingen · Xi’an Jiaotong University
Abstract
Serving systems choose a weight format for each model and deployment. A low-cost screen predicts the resulting KL divergence from the unquantized model's Fisher information using , the expansion underlying Hessian-aware and second-order post-training quantizers. We assess the reliability of this screen in GPU batches with channel-wise int8, per-tensor fp8 and group-wise int4 on Qwen3-8B, Qwen3-4B, OLMo-2-7B, Qwen3-32B and Qwen3-30B-A3B. Relative predictions hold across a 16-fold range of perturbation scales, int4 group sizes and rounding modes, while absolute predictions are heavy-tailed. A trimmed average brings 10 of 12 cells on Qwen3-4B and OLMo-2 within 15% of measured KL. The MoE model exhibits nonquadratic response at small perturbations, with int8 KL scaling as . Computing per-sequence gradients in bf16 can cause fourfold prediction errors. We trace these errors to arithmetic in the lower half of the backward pass and eliminate them by using fp32 there. Batch-invariant kernels run at 0.55 times the throughput, while changes in reduction order introduce a floor of 0.17-0.22 nats per sequence. We build a screener with one streamed fp32 Fisher pass, trimmed tensor-block predictions with a band, a flag for aligned tensors and per-layer format allocation. On the unseen Mistral-7B-v0.3 family, all six predictions fall within the band, all three KL budgets are met, and vLLM throughput is predicted within 0.4%.
Figures & tables
| Model | Scheme | Blow-ups | ||||
|---|---|---|---|---|---|---|
| fp32 | bf16 | fp32 | bf16 | % (max) | ||
| Qwen3-4B | int8 ch. | 1.13 | 0.57 | 0.99 | 0.98 | 0.2 (249) |
| Qwen3-4B | fp8 tensor | 0.95 | 0.44 | 0.95 | 0.95 | 0.2 (271) |
| Qwen3-4B | int4 g128 | 1.04 | 0.50 | 1.10 | 1.09 | 0.4 (247) |
| OLMo-2-7B | int8 ch. | 1.02 | 1.02 | 1.03 | 1.03 | 0.0 (5) |
| OLMo-2-7B | fp8 tensor | 0.99 | 0.99 | 1.00 | 0.99 | 0.0 (6) |
| cell | bits | KL | KL | tok/s | LAMBADA | |
|---|---|---|---|---|---|---|
| int8 | 8.00 | 0.115 | 0.127 | 1.11 | 4150 | 75.0 |
| fp8 | 8.00 | 0.459 | 0.473 | 1.03 | – | – |
| int4 | 4.12 | 10.8 | 12.1 | 1.12 | 4709 | 73.2 |
| 6.55 | 3.31 | 3.58 | 1.08 | 4343 (4343) | 74.6 | |
| 7.28 | 1.64 | 1.74 | 1.06 | 4226 (4244) | 74.5 | |
| 7.64 | 0.836 | 0.891 | 1.07 | 4180 (4196) | 74.7 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Batch (section) | Model(s) | Frozen prediction / criterion | Outcome |
|---|---|---|---|
| Scale, group size, rounding (§ 4 ) | Qwen3-8B | quadratic in ; group sizes ranked; SR/RN ratio | ratio 0.99–1.05 over ; 0.96–1.01; 2.25 vs. 2.28 (pass) |
| Reproduction of the earlier pipeline (§ 6 ) | Qwen3-8B | the over-prediction reproduces unchanged | (reproduced); same samples give the same numbers |
| Pre-flagged sequences (§ 6 ) | Qwen3-8B | dropping three sequences flagged before the run restores | (pass) |
| Where the error enters (§ 6 ) | Qwen3-8B | error confined to the weight-gradient products | 0/4 (fail): it lives in the residual-stream gradient and grows toward the input |
| Lower vs. upper half in fp32 (§ 6 ) | Qwen3-8B | the error is made below layer 16 | bottom error 0.017 with the lower half in fp32 vs. 0.96 with the upper half (pass) |
| Trimmed average on new models (§ 7 ) | Qwen3-4B, OLMo-2 | P1 ; P2 mean trim; P3 scheme ratios ; P4 blow-ups | P1 10/12; P2 11/12; P3 3/4; P4 6/6 (misses: OLMo-2 int4; Qwen3-4B blow-ups) |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers