QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncation

Lin-Fa Lee; Yi-Yu Chang; Kuo-Hei Yeh

cs.AIAug 14, 2026

QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncation

Authors: Lin-Fa Lee, Yi-Yu Chang, Kuo-Hei Yeh

Organizations: Department of Institute of Artificial Intelligence Innovation National Yang Ming Chiao Tung University Hsinchu, Taiwan

Abstract

Recent training-free post-training quantization methods restore model accuracy through closed-form residual compensation. To constrain additional model storage overhead, several existing methods gate layer selection by goodness-of-fit, retaining only those layers whose compensation yields a positive residual fit score and discarding the rest. In this paper, we show that, under the low-bit W4A4 setting, this gating mechanism fails to distinguish poorly predictable quantization error from numerical solver failure. Rank-deficient input activations yield severely ill-conditioned or numerically singular Gram matrices, causing the closed-form solver to become unstable and produce spuriously negative fit scores. Consequently, existing goodness-of-fit gates misclassify affected layers as uncompensable and discard them. Many of these discarded layers can nevertheless provide substantial error recovery when their compensation is computed using a numerically stable solver. To address this problem, we propose a parameter-free truncated pseudoinverse solver which removes collapsed directions prior to inversion. On ViT-B with the W4A4 setting, our training-free method achieves 81.42% top-1 accuracy, outperforming prior post-training methods and fine-tuning-based baselines. Combined with joint low-rank and quantization compression, the proposed method reaches a deployable operating point of 80.26% accuracy at 54.7 MB, providing a well-balanced trade-off between model size and accuracy.

Figures & tables

Figure 3: Progressive accuracy improvements across the three stages of our method

Method	Top 1 (%)	Size (MB)	Training free
RepQ-ViT (baseline, uncompensated)	68.5	44.9	—
QwT	76.3	59.1	✓
QwT*	78.5	59.1	$\times$ (Fine tuning)
QwT-v2	75.6	45.6	✓
IGQ-ViT	79.3	—	—
Ours (Compressed)	80.26 $\pm$ 0.11	54.7	✓

Table 1: Comparison of main results on ViT-B / W4A4 / ImageNet.

Model	Arch.	Block level (QwT)	Direct Inverse	Ours (stable)	Neg- $R^{2}$ / Max Cond.	Failure Mode of Direct Inverse
ViT-B	ViT	76.43	79.52 † (collapse 3/5)	pinv 81.42 $\pm$ 0.06	5 / $2.3\times 10^{9}$	Divergent parameters; accuracy collapses to near random
Swin-T	Swin	74.72	78.40 $\pm$ 0.63	pinv 78.90 $\pm$ 0.14	2–7 / $3.4\times 10^{10}$	Unstable solution; excessively high variance
DeiT-T	ViT	61.40	64.66 $\pm$ 0.16	pinv 64.76 $\pm$ 0.13	1–2 / $6.1\times 10^{9}$	A few layers judged failed and skipped
ResNet-50	CNN	62.5 ‡	Fails (3/3)	ridge 68.72 / pinv 66.63	singular / $\infty$	Gram exactly singular; inversion undefined and fails

Table 2: Cross architecture numerical vulnerability. Ill conditioning is universal but manifests as distinct failure modes. † ViT-B direct inverse is reported at seed 0; averaging is uninformative as 3/5 seeds collapse to near random. Condition numbers indicate severity; collapse is driven by rank deficiency rather than large condition number alone (Appendix A). ‡ ResNet-50 uses Percentile quantization (its QwT reference), not RepQ-ViT, so its baseline is not directly comparable to the transformer rows.

Method	Top 1 (%)	Collapsed Seeds	Layers with Negative $R^{2}$
OLS (direct inverse)	79.52	3 / 5	5
Truncated pseudo inverse	81.42 $\pm$ 0.06	0 / 5	0
Ridge (Tikhonov)	81.64 $\pm$ 0.04	0 / 5	0

Table 3: Comparison of solving methods for compensation.

Figure 5: (a)Severity of ill conditioning across architectures (b)Accuracy comparison before and after stabilization.

Rank / Bits	16-bit	8-bit	4-bit
$r=256$	81.14 / 119.0	81.16 / 82.3	80.79 / 63.9
$r=128$	80.79 / 82.4	80.67 / 64.0	80.50 / 54.7
$r=64$	80.34 / 64.0	80.37 / 54.8	79.87 / 50.1

Table 4: Relationship between rank and quantization bits. Performance is reported as Top-1 Accuracy (%) / Size (MB).

Figure 6: Trade off plot between accuracy and size

Quantization Error Compensation

See all 16 papers

QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
Aug 14, 2026Vincent Counathe, Ben Athiwaratkun, Christopher De Sa +1LLM Quantization Quantization-Aware Training
Cornell University · Together AI
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
Jun 8, 2026Hanyang Li, Jianhao Ma, Ying CuiNeural Network Optimization Straight-Through Estimator
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
May 31, 2026Shihao Zhang, Rayan SaabLLM Quantization Low-Rank Adaptation
Department of Mathematics, University of California San Diego

Aug 14, 2026J/K move · Enter open · S save

QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction

Vincent Counathe, Ben Athiwaratkun, Christopher De Sa, Tianyi Zhang

Cornell University · Together AI

LLM Quantization Quantization-Aware Training Low-Bit Quantization

Open paper AskPDF arXiv

As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT has a structural mismatch: gradient updates are applied to latent full-precision weights, while the loss and gradients are computed on lossy reconstructions of those weights. This mismatch can lead to suboptimal training trajectories and a higher loss floor. Second-order PTQ methods address a similar problem by minimizing loss-aware reconstruction error, but applying such expensive reconstruction repeatedly during QAT as the weights evolve is impractical. We introduce QUASAR, a QAT method that brings lightweight, loss-aware reconstruction into the training loop. At each training step, QUASAR reconstructs the latent weights by searching over a small set of clipping ranges and fitting dequantization parameters through saliency-weighted least squares. We use an exponential moving average of squared gradients as the per-parameter saliency signal. Our theoretical analysis shows that optimizing QUASAR's reconstruction objective tightens both the convergence and final-loss bounds of QAT. We evaluate QUASAR across four model families and across INT4, INT3, INT2, and NVFP4 quantization formats. QUASAR consistently achieves lower training and evaluation loss than competitive QAT methods and outperforms QAT and PTQ baselines on downstream benchmarks. At INT2, QUASAR improves average accuracy over the best QAT baseline by 13.3 points with quantization-aware distillation and by 10.9 points with QAT on mathematical reasoning data. Notably, after distillation on only about 600M tokens, QUASAR's INT4 Gemma-4 E4B checkpoint outperforms the corresponding QAT checkpoint released by Google, with 66% lower KL divergence and 1.8 points higher average accuracy.

Abstract

Quantization Error Compensation

Explore similar work

QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation