GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Peilin Sun; Guang Liang; Jin Tong; Jianxin Wu

cs.CVSep 28, 2026

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Authors: Peilin Sun, Guang Liang, Jin Tong, Jianxin Wu

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China

Abstract

Post-training quantization (PTQ) efficiently compresses Vision Transformers (ViTs) without retraining, yet suffers severe accuracy degradation at low bit-widths. Existing optimization-based PTQ methods guide block reconstruction via either soft logits or second-order Hessian proxies. Logit supervision is prone to overfitting on limited calibration data, while Hessian approximations incur structural truncation errors. To address these limitations, we propose \textbf{GLF-Q}, a novel PTQ framework guided by Global-Local Feature alignment. GLF-Q propagates quantized block outputs through downstream full-precision layers to align penultimate-layer representations under local output regularization, providing downstream feature supervision without explicitly approximating the Hessian or using a Taylor expansion. Furthermore, offline Hadamard transformations are introduced with zero runtime overhead to disperse activation outliers across channels, effectively contracting dynamic ranges and reducing quantization errors. Meanwhile, optimizing this loss via a Straight-Through Estimator (STE) achieves rapid convergence, bypassing continuous relaxation rounding formulations such as AdaRound. Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification. In addition, GLF-Q exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deployment.

Figures & tables

Method	Opt.	Spec.	W/A	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B
Full-Prec	-	-	32/32	81.39	84.54	72.21	79.85	81.80	83.23	85.27
PTQ4ViT ( Yuan et al., 2022 )	$\times$	$\checkmark$	3/3	0.10	0.10	3.50	0.10	31.06	28.69	20.13
RepQ-ViT ( Li et al., 2023 )	$\times$	$\checkmark$	3/3	0.10	0.10	0.10	0.10	0.10	0.10	0.10
AdaLog ( Wu et al., 2024 )	$\times$	$\checkmark$	3/3	13.88	37.91	31.56	24.47	57.47	64.41	69.75
I&S-ViT ( Zhong et al., 2023 )	$\checkmark$	$\checkmark$	3/3	45.16	63.77	41.52	55.78	73.30	74.20	69.30
DopQ-ViT ( Yang et al., 2024 )	$\checkmark$	$\checkmark$	3/3	54.72	65.76	44.71	59.26	74.91	74.77	69.63

Table 1: Top-1 accuracy comparison across representative Vision Transformers on ImageNet. “*” denotes results reimplemented using the FIMA-Q framework, as QDrop and PD-Quant were originally designed for CNNs. “Opt.” indicates whether a PTQ method is optimization-based (

\checkmark

) or calibration-only (

\times

). “Spec.” indicates reliance on a specialized non-uniform quantizer (

\checkmark

) versus a standard uniform quantizer (

\times

). The best results within each bit-width setting are highlighted in boldface .

Method	Opt.	Spec.	W/A	Mask R-CNN				Cascade Mask R-CNN
				Swin-T		Swin-S		Swin-T		Swin-S
				$\text{AP}^{b}$	$\text{AP}^{m}$	$\text{AP}^{b}$	$\text{AP}^{m}$	$\text{AP}^{b}$	$\text{AP}^{m}$	$\text{AP}^{b}$	$\text{AP}^{m}$
Full-Precision	-	-	32/32	46.0	41.6	48.5	43.3	50.4	43.7	51.9	45.0
PTQ4ViT ( Yuan et al., 2022 )	$\times$	$\checkmark$	4/4	6.9	7.0	26.7	26.6	14.7	13.5	0.5	0.5
APQ-ViT ( Ding et al., 2022 )	$\times$	$\checkmark$	4/4	23.7	22.6	44.7	40.1	27.2	24.4	47.7	41.1
RepQ-ViT ( Li et al., 2023 )	$\times$	$\checkmark$	4/4	36.1	36.0	44.2	40.2	47.0	41.1	49.3	43.1

Table 2: Object detection and instance segmentation performance on COCO with Mask R-CNN and Cascade Mask R-CNN under 4-bit quantization.

Setting	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B
LF	60.48	77.06	53.16	70.16	76.76	76.66	78.46
DPLR-FIM ( Wu et al., 2025a )	63.83	77.15	54.94	70.67	76.60	77.30	79.19
Logits + LF ( Liu et al., 2023 )	65.16	76.89	55.26	70.71	76.87	76.80	78.58
GF	66.09	77.17	54.06	70.41	76.69	77.75	80.30
GLF (Ours)	67.40	78.18	56.48	71.55	77.20	78.10	80.45
FIMA-Q + MR + Had. †	66.88	76.13	55.29	69.81	76.24	77.37	79.42

Table 3: Top-1 accuracy comparison (%) with different quantization losses and an additional baseline on ImageNet under 3-bit quantization.

\dagger

: FIMA-Q + MR + Hadamard with AdaRound, 20k iterations.

Method	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B
GLF only	60.37	72.50	55.28	68.75	75.79	76.92	79.31
GLF + MR	63.53	73.60	55.24	69.00	75.79	77.96	80.09
GLF + Hadamard	65.04	77.16	55.69	71.30	76.91	77.63	79.81
GLF-Q	67.40	78.18	56.48	71.55	77.20	78.10	80.45

Table 4: Top-1 accuracy comparison (%) with different framework components on ImageNet under 3-bit quantization. “GLF” denotes Global-Local Feature loss, “MR” denotes MLP Reconstruction, and “Hadamard” denotes Offline Hadamard Transformation.

Method	3-bit			4-bit
Method	ViT-S	DeiT-S	Swin-S	ViT-S	DeiT-S	Swin-S
FIMA-Q ( Wu et al., 2025a )	41.86	36.03	69.95	73.37	75.89	80.72
DPLR-FIM †	47.80	65.58	70.77	73.88	76.02	80.51
GLF-Q (Ours)	58.72	68.00	74.71	75.34	76.46	81.07

Table 5: ImageNet top-1 accuracy (%) with SUN397 calibration.

\dagger

: DPLR-FIM in our framework.

Method	Latency			Top-1 Accuracy
Method	ViT-S	DeiT-S	Swin-S	ViT-S	DeiT-S	Swin-S
FIMA-Q ( Wu et al., 2025a )	2.95	2.95	6.83	79.19	78.72	83.06
GLF-Q (Ours)	2.50	2.49	5.88	80.35	79.07	83.08

Table 6: Comparison of inference latency and Top-1 accuracy under 8-bit quantization. Latency (ms) is measured on a single NVIDIA H800 GPU with a batch size of 64 using NVIDIA TensorRT ( NVIDIA Corporation, 2024 ) , and Top-1 accuracy is evaluated on ImageNet.

Method	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B
FIMA-Q ( Wu et al., 2025a )	48	87	44	49	88	115	135
LS-ViT ( Hwang et al., 2026 )	40	56	41	40	56	101	104
GLF-Q (Ours)	21	34	19	20	34	57	63

Table 7: Comparison of training time (minutes) on ImageNet under 3-bit quantization.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Network Position	Parameter Transformation Rule
Input Patch Embedding	$E_{\mathrm{patch}}\to E_{\mathrm{patch}}R_{1}\,,\quad b_{E}\to b_{E}R_{1}$
Class Token & Positional Embeddings	$t\to tR_{1}\,,\quad p\to pR_{1}$
QKV Projection & MLP First Layer	$W_{\mathrm{in}}\to R_{1}^{\top}W_{\mathrm{in}}\,,\quad b_{\mathrm{in}}\text{ (unchanged)}$
Attention Output Projection & MLP Second Layer	$W_{\mathrm{out}}\to W_{\mathrm{out}}R_{1}\,,\quad b_{\mathrm{out}}\to b_{\mathrm{out}}R_{1}$
Final Classification Head	$W_{\mathrm{head}}\to R_{1}^{\top}W_{\mathrm{head}}\,,\quad b_{\mathrm{head}}\text{ (unchanged)}$

Appendix

Table 8: Parameter transformation rules for the centered and affine-folded Vision Transformer.

	Logits + LF		GLF
Model	Calibration	Validation	Calibration	Validation
ViT-S	86.91	65.16	87.70	67.40
ViT-B	91.11	76.89	91.11	78.18
DeiT-T	77.05	55.26	76.66	56.48
DeiT-S	90.04	70.71	89.65	71.55
DeiT-B	96.00	76.87	96.00	77.20

Appendix

Table 9: Top-1 accuracy (%) on the calibration set and ImageNet validation set under 3-bit quantization.

W/A	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B
3/3	$67.34\pm 0.12$	$78.21\pm 0.04$	$56.17\pm 0.27$	$71.67\pm 0.10$	$77.22\pm 0.10$	$78.07\pm 0.06$	$80.44\pm 0.03$
4/4	$77.69\pm 0.15$	$83.12\pm 0.10$	$67.22\pm 0.16$	$77.40\pm 0.06$	$80.51\pm 0.04$	$82.05\pm 0.08$	$83.81\pm 0.01$

Appendix

Table 10: Multi-seed top-1 accuracy (%) of GLF-Q on ImageNet under 3-bit and 4-bit quantization, reported as mean

\pm

standard deviation over three random seeds.

$\lambda$	ViT-S	ViT-B	DeiT-T	DeiT-S	DeiT-B	Swin-S	Swin-B	Mean
1	67.53	78.29	56.31	71.47	77.33	78.22	80.49	72.81
2	67.40	78.18	56.48	71.55	77.20	78.10	80.45	72.77
4	67.48	78.18	56.61	71.59	77.22	78.14	80.35	72.80

Appendix

Table 11: Top-1 accuracy (%) of GLF-Q with different regularization coefficients under 3-bit quantization on ImageNet.

Explore similar work

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training 4-Bit Quantization
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China
ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
Jun 20, 2026Changjun Li, Runqing Jiang, Lian Xu +3Vision Transformer Activation Quantization
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-sen University, Shenzhen, China · Computer Science and Software Engineering, The University of Western Australia, Perth, Australia · University of Oxford, Oxford, United Kingdom
MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers
Jul 30, 2026Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar FarukMixed-Precision Quantization Vision Transformer
Department of Computer Science and Engineering Khulna University of Engineering & Technology (KUET) Khulna 9203, Bangladesh

Jun 13, 2025, cs.CVJ/K move · Enter open · S save

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

Guang Liang, Xinyao Liu, Jianxin Wu

State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China

Quantization-Aware Training 4-Bit Quantization Efficient ViTs Activation Quantization Neural Network Quantization

Open paper AskPDF arXiv

Vision Transformers (ViTs) are essential in computer vision but are computationally intensive, too. Model quantization, particularly to low bit-widths like 4-bit, aims to alleviate this difficulty, yet existing Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) methods exhibit significant limitations. PTQ often incurs substantial accuracy drop, while QAT achieves high accuracy but suffers from prohibitive computational costs, limited generalization to downstream tasks, training instability, and lacking of open-source codebase. To address these challenges, this paper introduces General, Practical, and Lightning Quantization (GPLQ), a novel framework designed for efficient and effective ViT quantization. GPLQ is founded on two key empirical insights: the paramount importance of activation quantization and the necessity of preserving the model's original optimization basin'' to maintain generalization. Consequently, GPLQ employs a sequential activation-first, weights-later'' strategy. Stage 1 keeps weights in FP32 while quantizing activations with a feature mimicking loss in only 1 epoch to keep it stay in the same ``basin'', thereby preserving generalization. Stage 2 quantizes weights using a PTQ method. As a result, GPLQ is 100x faster than existing QAT methods, lowers memory footprint to levels even below FP32 training, and achieves 4-bit model performance that is highly competitive with FP32 models in terms of both accuracy on ImageNet and generalization to diverse downstream tasks, including fine-grained visual classification and object detection. We will release an easy-to-use open-source toolkit supporting multiple vision tasks.

Abstract

Explore similar work

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers

MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers