We show that robustness to post-training quantization (PTQ) is a transferable direction in weight space. We call this direction the quantization vector: extracted from a donor task by simple weight-space arithmetic, it can be used to patch a receiver model and improve post-PTQ Top-1 accuracy by up to 60 points in a 3-bit setting, without receiver-side quantization-aware training (QAT). Because the method requires no receiver training data, it provides a zero-shot, low-cost alternative to QAT for extremely low-bit deployment. Across four ViT scales and 22 image classification tasks, donor quantization vectors often yield substantial gains even when donor and receiver tasks differ markedly. We further prove rigorously that quantization vectors are well-defined and do not suffer from reparameterization symmetries, and provide a local geometric account of their effect. Together, these results suggest that quantization robustness can be partially isolated, reused, and transferred through simple weight-space algebra.
Figures & tables
Figure 1: Zero-shot QV patching. A donor quantization vector ρD:−θD,QAT−θD , extracted as the weight-space displacement between a standard fine-tuned donor checkpoint and its QAT counterpart, is added to a receiver checkpoint to obtain the patched model θR←D=θR+λρD . The plot is not to scale but uses true numbers; it illustrates the intended operating regime of our method, namely improving low-bit accuracy over PTQ without paying the full receiver-side cost of QAT.
Figure 2: Geometric view of donor patching. The blue vector ρR is the receiver’s own backbone QV ( unknown in our setting), the green ray is the donor direction ρD , and the red vector λ⋆ρD is the orthogonal projection of ρR onto that donor line. Proposition 1 states that the fraction of receiver-side QAT gain recovered by this best donor patch is exactly cos2γ , where γ is the angle between ρD and ρR . Illustration in whitened coordinates where the Hessian HR defines the local geometry.
Figure 3: QV patching across vision and text classifiers. Receiver Top-1 accuracy under 3-bit PTQ for vanilla fine-tuned checkpoints (FT+PTQ), QV-patched checkpoints (QV Patching+PTQ), and receiver-side QAT checkpoints (QAT+PTQ) on ViT-B/16 and Qwen3-Embedding .
Figure 4: Cross-task QV transfer for vision (left) and text (right) classifiers. Top-1 accuracy change from patching receiver r with donor d quantization vector, relative to vanilla 3-bit PTQ.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Hugging Face identifier
Stanford Cars [ 31 ]
tanganke/stanford_cars
CIFAR-10 [ 32 ]
uoft-cs/cifar10
CIFAR-100 [ 32 ]
uoft-cs/cifar100
DTD [ 9 ]
tanganke/dtd
EMNIST [ 13 ]
tanganke/emnist_letters
EuroSAT [ 24 ]
tanganke/eurosat
Appendix
Table 1: Vision datasets. Dataset names, citations, and Hugging Face identifiers used in the vision experiments.
Dataset
Hugging Face identifier
Emotion [ 57 ]
mteb/emotion
IMDB [ 41 ]
mteb/imdb
Banking77 [ 5 ]
mteb/banking77
AmazonReviewsClassification [ 29 ]
mteb/AmazonReviewsClassification
AmazonCounterfactualClassification [ 52 ]
mteb/amazon_counterfactual
MassiveIntentClassification [ 19 ]
mteb/amazon_massive_intent
Appendix
Table 2: Text datasets. MTEB classification datasets, citations, and Hugging Face identifiers used in the text experiments.
Head regime
Framework
Model identifier
Pretrained version
CLIP-style
open_clip
ViT-B/16
laion2b_s34b_b88k
CLIP-style
open_clip
ViT-L/14
laion2b_s32b_b82k
CLIP-style
open_clip
ViT-H/14
laion2b_s32b_b79k
Learned head
timm
deit3_base_patch16_224
fb_in1k
Learned head
timm
deit3_large_patch16_224
fb_in1k
Learned head
timm
swin_base_patch4_window7_224
ms_in22k_ft_in1k
Appendix
Table 3: Vision model identifiers. Exact vision backbone identifiers, head regimes, frameworks, and pretrained versions used in the experiments.
Model
Hugging Face identifier
BERT-base
google-bert/BERT-base-uncased
BERT-large
google-bert/BERT-large-uncased
EmbeddingGemma-300M
google/EmbeddingGemma-300M
Qwen3-Embedding-0.6B
Qwen/Qwen3-Embedding-0.6B
Appendix
Table 4: Text encoder identifiers. Exact Hugging Face identifiers used for learned-head text classification.
Figure 5: Comparison between accuracy of the full precision finetuned model (FT), the corresponding QAT and FP + PTQ, for the vision backbones used throughout the paper.
Figure 6: Comparison between accuracy of the full precision finetuned model (FT), the corresponding QAT and FP + PTQ, for the clip-style vision backbones used throughout the paper.
Figure 7: Comparison between accuracy of the full precision finetuned model, the corresponding QAT and FP + PTQ, for the text backbones used throughout the paper.
Figure 8: QV transferability for learned-head ViT-B/16 (vision) . Top-1 accuracy change ( Δ ) from patching receiver r with donor d quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( λ=1 ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude λ eliminates destructive interference while maximizing gains.
Figure 9: QV transferability for Qwen3-Embedding (text) . Top-1 accuracy change ( Δ ) from patching receiver r with donor d quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( λ=1 ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude λ eliminates destructive interference while maximizing gains.
Figure 10: QV transferability for vision (learned-head) . Top-1 accuracy change ( Δ ) from patching receiver r with donor d quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( λ=1 ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude λ eliminates destructive interference while maximizing gains.
Figure 11: QV transferability for vision (CLIP-style) . Top-1 accuracy change ( Δ ) from patching receiver r with donor d quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( λ=1 ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude λ eliminates destructive interference while maximizing gains.
Figure 12: QV transferability for text . Top-1 accuracy change ( Δ ) from patching receiver r with donor d quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( λ=1 ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude λ eliminates destructive interference while maximizing gains.
Post-training quantization (PTQ) converts a trained full-precision model into low-bit weights without task-level retraining, while quantization-aware training (QAT) incorporates quantization into the training loop. Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths; QAT is more expensive but can often recover the lost accuracy. We propose a unified geometric framework that explains both PTQ failure and QAT recovery. We model full-precision training as following a low-loss river inside a wider valley: a normal neighborhood of the river forms a nearly flat basin, while leaving this basin incurs a sharp loss increase. When the quantization grid is comparable to the basin width, local PTQ objectives, including rounding and Hessian-based second-order reconstruction, can select a high-loss deployed quantized point outside the basin even when nearby low-loss quantized points exist. In this regime, straight-through-estimator-based QAT has a useful bias: it evaluates gradients at the deployed quantized weights while updating latent full-precision weights, causing the gradient to sense the valley wall and acquire an inward component that steers subsequent quantized iterates back into the basin. We formalize this mechanism through a local landscape model, construct a geometric PTQ failure mode, and prove finite-time QAT recovery under local quantizer-compatibility assumptions. Experiments across vision and language models under multiple neural-network quantization schemes corroborate the predicted basin-crossing failure of PTQ and the corresponding recovery mechanism of QAT.
Hanyang Li, Jianhao Ma, Ying Cui
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.
Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6
University of Science and Technology of China · University College London · Shanghai University of Finance and Economics +5
We present a framework for end-to-end joint quantization of Vision Transformers trained on ImageNet for the purpose of image classification. Unlike prior post-training or block-wise reconstruction methods, we jointly optimize over the entire set of all layers and inter-block dependencies without any labeled data, scaling effectively with the number of samples and completing in just one hour on a single GPU for ViT-small. We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet and, to the best of our knowledge, the first PTQ results that maintain strong accuracy on ViT, DeiT, and Swin-T models under extremely low-bit settings (W1.58A8), demonstrating the potential for efficient edge deployment. Furthermore, we introduce a data-free calibration strategy that synthesizes diverse, label-free samples using Stable Diffusion Turbo guided by learned multi-mode prompts. By encouraging diversity in both the learned prompt embeddings and the generated image features, our data-free approach achieves performance on par with real-data ImageNet calibration and surpasses simple text-prompt baselines such as "a <adjective> photo of <adjective> <cls>".
Shile Li, Markus Karmann, Onay Urfalioglu
vivo Tech Research GmbH · Torr Vision Group, Universtiy of Oxford