Zero-Shot Quantization via Weight-Space Arithmetic

Daniele Solombrino; Antonio Andrea Gargiulo; Alessandro Zirilli; Luca Zhou; Adrian Robert Minut; Emanuele Rodolà

cs.CVApr 3, 2026

Zero-Shot Quantization via Weight-Space Arithmetic

Authors: Daniele Solombrino, Antonio Andrea Gargiulo, Alessandro Zirilli, Luca Zhou, Adrian Robert Minut, Emanuele Rodolà

Organizations: Sapienza University of Rome · Sapienza University of Rome / Paradigma

Abstract

We show that robustness to post-training quantization (PTQ) is a transferable direction in weight space. We call this direction the quantization vector: extracted from a donor task by simple weight-space arithmetic, it can be used to patch a receiver model and improve post-PTQ Top-1 accuracy by up to 60 points in a 3-bit setting, without receiver-side quantization-aware training (QAT). Because the method requires no receiver training data, it provides a zero-shot, low-cost alternative to QAT for extremely low-bit deployment. Across four ViT scales and 22 image classification tasks, donor quantization vectors often yield substantial gains even when donor and receiver tasks differ markedly. We further prove rigorously that quantization vectors are well-defined and do not suffer from reparameterization symmetries, and provide a local geometric account of their effect. Together, these results suggest that quantization robustness can be partially isolated, reused, and transferred through simple weight-space algebra.

Figures & tables

Figure 1: Zero-shot QV patching. A donor quantization vector $\rho_{\mathcal{D}}\coloneq\theta_{\mathcal{D},\text{QAT}}-\theta_{\mathcal{D}}$ , extracted as the weight-space displacement between a standard fine-tuned donor checkpoint and its QAT counterpart, is added to a receiver checkpoint to obtain the patched model $\theta_{\mathcal{R}\leftarrow\mathcal{D}}=\theta_{\mathcal{R}}+\lambda\rho_{\mathcal{D}}$ . The plot is not to scale but uses true numbers; it illustrates the intended operating regime of our method, namely improving low-bit accuracy over PTQ without paying the full receiver-side cost of QAT.

Figure 2: Geometric view of donor patching. The blue vector

{\rho}_{\mathcal{R}}

is the receiver’s own backbone QV ( unknown in our setting), the green ray is the donor direction

{\rho}_{\mathcal{D}}

, and the red vector

\lambda^{\star}{\rho}_{\mathcal{D}}

is the orthogonal projection of

{\rho}_{\mathcal{R}}

onto that donor line. Proposition 1 states that the fraction of receiver-side QAT gain recovered by this best donor patch is exactly

\cos^{2}\gamma

, where

\gamma

is the angle between

{\rho}_{\mathcal{D}}

and

{\rho}_{\mathcal{R}}

. Illustration in whitened coordinates where the Hessian

H_{\mathcal{R}}

defines the local geometry.

Figure 3: QV patching across vision and text classifiers. Receiver Top-1 accuracy under 3-bit PTQ for vanilla fine-tuned checkpoints (FT+PTQ), QV-patched checkpoints (QV Patching+PTQ), and receiver-side QAT checkpoints (QAT+PTQ) on ViT-B/16 and Qwen3-Embedding .

Figure 4: Cross-task QV transfer for vision (left) and text (right) classifiers. Top-1 accuracy change from patching receiver $r$ with donor $d$ quantization vector, relative to vanilla 3-bit PTQ.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Hugging Face identifier
Stanford Cars [ 31 ]	tanganke/stanford_cars
CIFAR-10 [ 32 ]	uoft-cs/cifar10
CIFAR-100 [ 32 ]	uoft-cs/cifar100
DTD [ 9 ]	tanganke/dtd
EMNIST [ 13 ]	tanganke/emnist_letters
EuroSAT [ 24 ]	tanganke/eurosat

Appendix

Table 1: Vision datasets. Dataset names, citations, and Hugging Face identifiers used in the vision experiments.

Dataset	Hugging Face identifier
Emotion [ 57 ]	mteb/emotion
IMDB [ 41 ]	mteb/imdb
Banking77 [ 5 ]	mteb/banking77
AmazonReviewsClassification [ 29 ]	mteb/AmazonReviewsClassification
AmazonCounterfactualClassification [ 52 ]	mteb/amazon_counterfactual
MassiveIntentClassification [ 19 ]	mteb/amazon_massive_intent

Appendix

Table 2: Text datasets. MTEB classification datasets, citations, and Hugging Face identifiers used in the text experiments.

Head regime	Framework	Model identifier	Pretrained version
CLIP-style	open_clip	ViT-B/16	laion2b_s34b_b88k
CLIP-style	open_clip	ViT-L/14	laion2b_s32b_b82k
CLIP-style	open_clip	ViT-H/14	laion2b_s32b_b79k
Learned head	timm	deit3_base_patch16_224	fb_in1k
Learned head	timm	deit3_large_patch16_224	fb_in1k
Learned head	timm	swin_base_patch4_window7_224	ms_in22k_ft_in1k

Appendix

Table 3: Vision model identifiers. Exact vision backbone identifiers, head regimes, frameworks, and pretrained versions used in the experiments.

Model	Hugging Face identifier
BERT-base	google-bert/BERT-base-uncased
BERT-large	google-bert/BERT-large-uncased
EmbeddingGemma-300M	google/EmbeddingGemma-300M
Qwen3-Embedding-0.6B	Qwen/Qwen3-Embedding-0.6B

Appendix

Table 4: Text encoder identifiers. Exact Hugging Face identifiers used for learned-head text classification.

Figure 5: Comparison between accuracy of the full precision finetuned model (FT), the corresponding QAT and FP + PTQ, for the vision backbones used throughout the paper.

Figure 6: Comparison between accuracy of the full precision finetuned model (FT), the corresponding QAT and FP + PTQ, for the clip-style vision backbones used throughout the paper.

Figure 7: Comparison between accuracy of the full precision finetuned model, the corresponding QAT and FP + PTQ, for the text backbones used throughout the paper.

Figure 8: QV transferability for learned-head ViT-B/16 (vision) . Top-1 accuracy change ( $\Delta$ ) from patching receiver $r$ with donor $d$ quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( $\lambda=1$ ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude $\lambda$ eliminates destructive interference while maximizing gains.

Figure 9: QV transferability for Qwen3-Embedding (text) . Top-1 accuracy change ( $\Delta$ ) from patching receiver $r$ with donor $d$ quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( $\lambda=1$ ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude $\lambda$ eliminates destructive interference while maximizing gains.

Figure 10: QV transferability for vision (learned-head) . Top-1 accuracy change ( $\Delta$ ) from patching receiver $r$ with donor $d$ quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( $\lambda=1$ ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude $\lambda$ eliminates destructive interference while maximizing gains.

Figure 11: QV transferability for vision (CLIP-style) . Top-1 accuracy change ( $\Delta$ ) from patching receiver $r$ with donor $d$ quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( $\lambda=1$ ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude $\lambda$ eliminates destructive interference while maximizing gains.

Figure 12: QV transferability for text . Top-1 accuracy change ( $\Delta$ ) from patching receiver $r$ with donor $d$ quantization vector, relative to baseline 3-bit PTQ. For each architecture, heatmaps on the left show transfer using a constant scaling factor ( $\lambda=1$ ), revealing both positive transfer and destructive interference. Heatmaps on the right demonstrate that modulating the magnitude $\lambda$ eliminates destructive interference while maximizing gains.

Quantization Robustness

See all 22 papers

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
Jun 8, 2026Hanyang Li, Jianhao Ma, Ying CuiNeural Network Optimization Straight-Through Estimator
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
Oct 8, 2026Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6LLM Quantization Post-Training Quantization
University of Science and Technology of China · University College London · Shanghai University of Finance and Economics +5
Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation
Feb 21, 2026Shile Li, Markus Karmann, Onay UrfaliogluPost-Training Quantization Neural Network Quantization
vivo Tech Research GmbH · Torr Vision Group, Universtiy of Oxford

Jun 8, 2026J/K move · Enter open · S save

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

Hanyang Li, Jianhao Ma, Ying Cui

Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania

Neural Network Optimization Straight-Through Estimator Quantization-Aware Training Neural Network Quantization Post-Training Quantization Low-Bit Quantization+1

Open paper AskPDF arXiv

Post-training quantization (PTQ) converts a trained full-precision model into low-bit weights without task-level retraining, while quantization-aware training (QAT) incorporates quantization into the training loop. Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths; QAT is more expensive but can often recover the lost accuracy. We propose a unified geometric framework that explains both PTQ failure and QAT recovery. We model full-precision training as following a low-loss river inside a wider valley: a normal neighborhood of the river forms a nearly flat basin, while leaving this basin incurs a sharp loss increase. When the quantization grid is comparable to the basin width, local PTQ objectives, including rounding and Hessian-based second-order reconstruction, can select a high-loss deployed quantized point outside the basin even when nearby low-loss quantized points exist. In this regime, straight-through-estimator-based QAT has a useful bias: it evaluates gradients at the deployed quantized weights while updating latent full-precision weights, causing the gradient to sense the valley wall and acquire an inward component that steers subsequent quantized iterates back into the basin. We formalize this mechanism through a local landscape model, construct a geometric PTQ failure mode, and prove finite-time QAT recovery under local quantizer-compatibility assumptions. Experiments across vision and language models under multiple neural-network quantization schemes corroborate the predicted basin-crossing failure of PTQ and the corresponding recovery mechanism of QAT.

Abstract

Quantization Robustness

Explore similar work

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation