RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

Mengyuan Fan; Bokai Huang; JiaMing Pan; Xiaokun Yuan; Peizhuang Cong; Zhewen Tan; Tong Yang

cs.CVOct 3, 2026

RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

Authors: Mengyuan Fan, Bokai Huang, JiaMing Pan, Xiaokun Yuan, Peizhuang Cong, Zhewen Tan, Tong Yang

Organizations: Peking University

Abstract

Vision Transformers (ViTs) achieve strong performance on image recognition and mobile vision applications, but their high-dimensional linear projections and attention computations still impose substantial storage and inference costs. Extremely low-bit quantization is a promising solution, yet ViTs often suffer severe accuracy degradation because conventional real-valued scalar codebooks are poorly matched to the directional geometry of Transformer projections. We present RPFQ-ViT, a Rotated Phase-Frame Quantization method that quantizes paired channels in two-dimensional phase planes, enabling low-bit codes to better preserve projection directions while recovering magnitude with lightweight scaling. RPFQ-ViT serves as a drop-in QAT replacement for nn.Linear and does not modify the standard real-valued attention, normalization, or activation computation graph. On ImageNet-1K, RPFQ-ViT-B/16 reaches 79.33% Top-1 / 94.48% Top-5 under W2/A4, Swin-T reaches 79.30% Top-1 / 94.79% Top-5 under W2/A8, and DeiT-S reaches 77.41% Top-1 / 93.11% Top-5 under W2/A8. Ablations, phase-geometry analysis, and direction-preservation metrics show that channel pairing, learnable rotation, phase-anchor learning, and residual phase refinement each improve quantization quality. We further deploy RPFQ-ViT image-classification models on native iOS and Android runtime stacks; with 2-bit packed weights, model size shrinks by roughly $5.4$ - $7.1\times$ relative to FP32 and end-to-end on-device latency drops by $1.4$ - $1.6\times$ . All ImageNet results trained in our codebase use a matched 300-epoch recipe and are reported as mean accuracies over three independent runs. These results show that RPFQ-ViT provides a favorable trade-off among accuracy, compression, and practical mobile deployment for extremely low-bit ViTs.

Figures & tables

Figure 1: Overview of RPFQ. Even-dimensional ViT linear layers are rewritten in paired complex coordinates, quantized with a learnable rotated phase frame plus least-squares magnitude fitting and residual refinement, and then mapped back to standard real-valued weights. Attention, normalization, and activation layers remain unchanged.

Figure 2: Phase-geometry view of low-bit quantization on half-split channel pairs from a ViT-B/16 linear layer. (a) Full-precision weights concentrate around a dominant direction. (b) Independent real-valued ternary quantization collapses directions toward the coordinate axes and induces large angular error. (c) RPFQ rotates a four-phase codebook toward the local dominant direction, substantially reducing the angular error under 2-bit quantization.

Backbone	Method	Source	Eff. weight bits	Epochs	Top-1	Top-5
ViT-B/16	FP32	reproduced	32	300	77.888	93.388
	Q-ViT	reproduced	2	300	62.378	83.840
	LSQ++KD	reproduced	2	300	75.10	92.18
	RPFQ	ours	2	300	79.326	94.476
DeiT-S	FP32	reproduced	32	300	78.364	93.738
	Q-ViT	reported	2	300	72.10	90.30

Table 1: ImageNet-1K main results under low-bit weight quantization. Gray rows denote full-precision references and blue rows denote our RPFQ results. For all non-reported rows, each entry is the mean over three independent runs using the same 300-epoch recipe and the same final-checkpoint evaluation rule for FP32 and low-bit methods trained in our codebase. Reported numbers are taken from the original papers whenever available; non-reported numbers use our training recipe. The “Eff. weight bits” column refers to the forward weight-code budget only, rather than collapsing method-specific side metadata into a single nominal bit count.

Variant	Pairing	Residual	Rotation	Anchors	Scale fit	Top-1	Top-5
Real scalar LSQ	$\times$	$\times$	$\times$	$\times$	real LSQ	74.89	91.28
Fixed phase frame	✓	$\times$	none	fixed	mean	76.56	92.83
+ residual refinement	✓	✓	none	fixed	mean	78.74	93.61
+ pre-aligned frame	✓	✓	pre-aligned	fixed	LS	78.79	93.78
RPFQ w/o anchor learning	✓	✓	learned	fixed	LS	79.06	94.21
RPFQ full	✓	✓	learned	learnable	LS	79.33	94.48

Table 2: Component ablation on ViT-B/16 W2/A4. All rows below “Real scalar LSQ” are internal paired-phase variants under the same quantized layer set and effective W2 weight-code budget. Pairing : deterministic paired phase planes; Residual : residual phase refinement; Rotation : phase-frame adaptation state; Anchors : phase codewords; Scale fit : magnitude estimation method. In the “Rotation” column, “none” is the fixed phase frame, “pre-aligned” is a coarse per-group pre-alignment with fixed canonical anchors, and “learned” is the end-to-end learned group-wise rotation used in RPFQ. Metadata are accounted separately using the per-group scalar rule in Section 3.3 .

Quantizer	Weight MSE $\downarrow$	Angular error $\downarrow$	Attention score MSE $\downarrow$	Feature cosine $\uparrow$	Top-1 $\uparrow$
Real LSQ W2/A4	1.00	1.00	1.00	0.931	74.89
Real ternary + KD	0.82	0.91	0.84	0.947	76.20
Fixed phase frame	0.63	0.57	0.61	0.971	78.74
RPFQ w/o residual	0.58	0.51	0.55	0.976	78.90
RPFQ full	0.49	0.43	0.46	0.983	79.33

Table 3: Direction-preservation analysis on ViT-B/16 W2/A4. All phase-based rows use the same paired-phase layer set and effective W2 weight-code budget; only frame adaptation and residual refinement differ. Weight MSE, angular error, and attention-score MSE are normalized to Real LSQ W2/A4; lower is better for the first three columns, while higher is better for feature cosine and Top-1.

Hardware profile	Runtime	Model	W/A	Size (MB)	Load (ms)	Latency p50 / p90 (ms)	Speedup
Recent Apple A-series iOS phone	MLX Swift	ViT-B/16 FP32	32/32	338.6	742	86.4 / 95.1	1.00 $\times$
	MLX Swift	RPFQ-ViT-B/16	2/4	47.7	213	54.9 / 61.3	1.57 $\times$
	MLX Swift	Swin-T FP32	32/32	110.2	318	39.8 / 44.5	1.00 $\times$
	MLX Swift	RPFQ-Swin-T	2/8	20.4	91	27.4 / 31.0	1.45 $\times$
Recent Android flagship phone	JNI/C++	ViT-B/16 FP32	32/32	338.6	918	123.6 / 139.2	1.00 $\times$
	JNI/C++	RPFQ-ViT-B/16	2/4	47.7	302	78.1 / 88.4	1.58 $\times$

Table 4: Native iOS and Android deployment results. “W/A” denotes the quantization setting of the source checkpoint; weights remain packed at 2 bits while the remaining operators run in floating point. “Size” is the exported model container, and latency is measured end-to-end after warm-up with batch size 1.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Comparison	Ours 95% CI	Baseline 95% CI	Gap (pt)	Gap 95% CI (pt)
ViT-B/16	RPFQ vs FP32	[78.97, 79.68]	[77.52, 78.25]	+1.44	[0.93, 1.95]
ViT-B/16	RPFQ vs LSQ++KD	[78.97, 79.68]	[74.72, 75.48]	+4.23	[3.71, 4.75]
DeiT-S	RPFQ vs OFQ	[77.04, 77.78]	[75.34, 76.10]	+1.69	[1.17, 2.21]
Swin-T	RPFQ vs OFQ	[78.94, 79.65]	[78.16, 78.88]	+0.78	[0.27, 1.28]

Appendix

Table 5: Conservative 95% confidence intervals for Top-1 comparisons supporting the main claims. Intervals quantify finite-sample uncertainty on the fixed 50,000-image ImageNet-1K validation set; they do not replace the three-seed averaging protocol, but they show that the principal accuracy gaps remain separated from zero even under a conservative approximation.

Pairing rule	Layout-preserving	Top-1	Top-5
Half-split $(j,j+m)$	yes	79.33	94.48
Adjacent $(2j{-}1,2j)$	yes	79.08	94.24
Random fixed pairing	no	78.81	94.05

Appendix

Table 6: Pairing-rule ablation on ViT-B/16 W2/A4 with the full RPFQ quantizer. All rows use the same training recipe, quantized layer set, and effective W2 code budget; only the deterministic channel-pairing rule changes. For the random row, we report the mean over three fixed random pairings.

Figure 3: Training dynamics on ViT-B/16 W2/A4. Left: validation Top-1 curves. Right: quantization-state flip rate. RPFQ converges to the best accuracy while maintaining a lower late-stage oscillation rate than the fixed phase frame and real-valued variants.

Figure 4: Last-layer [CLS] attention maps on three ImageNet validation images (rows: Hummingbird, Italian Greyhound, Acoustic Guitar). Columns show the input, FP32 reference, Real LSQ W2, Fixed phase frame W2, and RPFQ W2; numbers in parentheses are the corresponding ImageNet-1K Top-1 accuracies. The annotated attn-sim badges are shown only on the non-FP32 columns and report cosine similarity to the FP32 reference map; the FP32 column is labeled “reference” and its self-similarity is trivially 1.0, so no score is displayed there. RPFQ yields the closest match (0.98–0.99), consistent with the directional metrics in Table 3 .

Method	Backbone / setting	Protocol	Aux. branch	Iter (s)	300-epoch total
RPFQ	ViT-B/16 W2/A4	single-stage QAT	none	0.223	46.55 h / 1.94 d
RPFQ	DeiT-S W2/A8	single-stage QAT	none	0.191	39.75 h / 1.66 d
RPFQ	Swin-T W2/A8	single-stage QAT	none	0.216	45.06 h / 1.88 d
Q-ViT	ViT-B/16 W2/A4	single-stage QAT	teacher + DGD attention distill.	0.768	160.21 h / 6.68 d
OFQ	DeiT-S / Swin-T (no ViT-B recipe)	QAT + CGA fine-tune	teacher + QKR + CGA	–	$\geq 600$ epochs

Appendix

Table 7: Training efficiency on ImageNet-1K with 8

\times

H800-80GB. RPFQ numbers are parsed from full training logs. Q-ViT runtimes are profiled from its official codebase under the same global batch size and projected to 300 epochs. For OFQ, we report only the schedule-level lower bound implied by its public two-stage protocol because no ViT-B/16 runtime is available.

Explore similar work

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training 4-Bit Quantization
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
Sep 28, 2026Peilin Sun, Guang Liang, Jin Tong +1Post-Training Quantization Low-Bit Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation
Feb 21, 2026Shile Li, Markus Karmann, Onay UrfaliogluPost-Training Quantization Neural Network Quantization
vivo Tech Research GmbH · Torr Vision Group, Universtiy of Oxford

Jun 13, 2025J/K move · Enter open · S save

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

Guang Liang, Xinyao Liu, Jianxin Wu

State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China

Quantization-Aware Training 4-Bit Quantization Efficient ViTs Activation Quantization Neural Network Quantization

Open paper AskPDF arXiv

Vision Transformers (ViTs) are essential in computer vision but are computationally intensive, too. Model quantization, particularly to low bit-widths like 4-bit, aims to alleviate this difficulty, yet existing Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) methods exhibit significant limitations. PTQ often incurs substantial accuracy drop, while QAT achieves high accuracy but suffers from prohibitive computational costs, limited generalization to downstream tasks, training instability, and lacking of open-source codebase. To address these challenges, this paper introduces General, Practical, and Lightning Quantization (GPLQ), a novel framework designed for efficient and effective ViT quantization. GPLQ is founded on two key empirical insights: the paramount importance of activation quantization and the necessity of preserving the model's original optimization basin'' to maintain generalization. Consequently, GPLQ employs a sequential activation-first, weights-later'' strategy. Stage 1 keeps weights in FP32 while quantizing activations with a feature mimicking loss in only 1 epoch to keep it stay in the same ``basin'', thereby preserving generalization. Stage 2 quantizes weights using a PTQ method. As a result, GPLQ is 100x faster than existing QAT methods, lowers memory footprint to levels even below FP32 training, and achieves 4-bit model performance that is highly competitive with FP32 models in terms of both accuracy on ImageNet and generalization to diverse downstream tasks, including fine-grained visual classification and object detection. We will release an easy-to-use open-source toolkit supporting multiple vision tasks.

Abstract

Explore similar work

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation