Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation

Shile Li; Markus Karmann; Onay Urfalioglu

cs.CVFeb 21, 2026

Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation

Authors: Shile Li, Markus Karmann, Onay Urfalioglu

Organizations: vivo Tech Research GmbH · Torr Vision Group, Universtiy of Oxford

Abstract

We present a framework for end-to-end joint quantization of Vision Transformers trained on ImageNet for the purpose of image classification. Unlike prior post-training or block-wise reconstruction methods, we jointly optimize over the entire set of all layers and inter-block dependencies without any labeled data, scaling effectively with the number of samples and completing in just one hour on a single GPU for ViT-small. We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet and, to the best of our knowledge, the first PTQ results that maintain strong accuracy on ViT, DeiT, and Swin-T models under extremely low-bit settings (W1.58A8), demonstrating the potential for efficient edge deployment. Furthermore, we introduce a data-free calibration strategy that synthesizes diverse, label-free samples using Stable Diffusion Turbo guided by learned multi-mode prompts. By encouraging diversity in both the learned prompt embeddings and the generated image features, our data-free approach achieves performance on par with real-data ImageNet calibration and surpasses simple text-prompt baselines such as "a <adjective> photo of <adjective> <cls>".

Figures & tables

Panel of Figure 1 — Figure 1: Our PTQ framework also supports a fully data-free mode using image synthesis via Stable Diffusion Turbo. For each ImageNet class ( kite , tench , and mountain bike ), top: images from simple text prompts show limited diversity and occasional semantic errors (e.g., kite as a toy rather than a bird). bottom: our learned multi-mode prompts (4 of 20 per class shown) generate semantically correct and diverse samples in layout, background, and style. These synthetic images are used for calibration in our data-free quantization pipeline.

Figure 2: Overview of the end-to-end quantization pipeline

Figure 3: Overview of the data-free multi-prompt learning pipeline. For each class (e.g., “tench”), multiple learned prompts are encoded by a frozen CLIP text encoder and used by Stable Diffusion-Turbo to synthesize diverse images under shared latent noise. A frozen ViT classifier provides supervision via classification loss, while orthogonality and variance losses encourage semantic and visual diversity.

Figure 4: Accuracy vs. calibration size for ViT-S (W4A4): comparison with FIMA-Q.

Figure 5: Scaling trends across models and bit settings. Performance gains starts to diminish beyond 10k calibration samples.

Method	ViT		DeiT		Swin
Method	S	B	S	B	S	B
Full-Precision	81.39	84.54	79.85	81.80	83.23	85.27
W1.58A8
ReQ-Vit [ 14 ]	0.12	0.13	0.11	0.11	0.09	0.09
FIMA-Q [ 26 ]	4.84	45.55	33.93	57.75	38.13	48.10
Ours (Synth)	63.71	76.58	67.06	77.18	73.06	75.51

Table 1: Top-1 accuracy (%) on ImageNet-1K for various backbones and bit-widths. Real: 10k real calibration images. Synth: 100k synthetic images generated via our learned multi-mode prompts. Best results are in bold , second best in underlined .

Figure 6: t-SNE visualization of ViT-Small feature embeddings. Each plot shows 100 samples per class for kite , tench , and mountain bike . Green: real ImageNet images. Red: synthetic images from raw text prompts. Blue: synthetic images from our learned multi-mode prompts. Our method produces broader, multi-cluster feature distributions closer to the real-image distribution, suggesting improved semantic coverage and feature diversity.

Figure 7: PTQ accuracy comparison between raw and learned prompts. Each group shows top-1 accuracy (%) on ImageNet for ViT-Small under three bit-width settings (W1.58A8, W3A3, W4A4). Learned prompts consistently outperform raw text prompts across all calibration sizes. The gain is most pronounced in low-bit regimes (e.g., +3.6% at W1.58A8), indicating that improved semantic coverage and feature diversity from multi-mode prompt learning lead to higher-quality synthetic calibration data.

Explore similar work

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
Sep 28, 2026Peilin Sun, Guang Liang, Jin Tong +1Post-Training Quantization Low-Bit Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
Jun 20, 2026Changjun Li, Runqing Jiang, Lian Xu +3Vision Transformer Activation Quantization
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-sen University, Shenzhen, China · Computer Science and Software Engineering, The University of Western Australia, Perth, Australia · University of Oxford, Oxford, United Kingdom
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training 4-Bit Quantization
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China

Sep 28, 2026J/K move · Enter open · S save

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Peilin Sun, Guang Liang, Jin Tong, Jianxin Wu

State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China

Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) efficiently compresses Vision Transformers (ViTs) without retraining, yet suffers severe accuracy degradation at low bit-widths. Existing optimization-based PTQ methods guide block reconstruction via either soft logits or second-order Hessian proxies. Logit supervision is prone to overfitting on limited calibration data, while Hessian approximations incur structural truncation errors. To address these limitations, we propose GLF-Q, a novel PTQ framework guided by Global-Local Feature alignment. GLF-Q propagates quantized block outputs through downstream full-precision layers to align penultimate-layer representations under local output regularization, providing downstream feature supervision without explicitly approximating the Hessian or using a Taylor expansion. Furthermore, offline Hadamard transformations are introduced with zero runtime overhead to disperse activation outliers across channels, effectively contracting dynamic ranges and reducing quantization errors. Meanwhile, optimizing this loss via a Straight-Through Estimator (STE) achieves rapid convergence, bypassing continuous relaxation rounding formulations such as AdaRound. Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification. In addition, GLF-Q exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deployment.

Abstract

Explore similar work

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers