Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

Wang Zixian

cs.CVApr 8, 2026

Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

Authors: Wang Zixian

Organizations: China Mobile Communications Group Shandong Co., Ltd.

Abstract

Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation. To this end, we propose Orthogonal Quadratic Complements (OQC), which construct a low-rank quadratic auxiliary branch and explicitly project it onto the orthogonal complement of the main branch before injection. We further study an efficient low-rank realization (OQC-LR) and gated extensions (OQC-static and OQC-dynamic). Under a parameter-matched Deep-ViT and CIFAR-100 protocol with a fixed penultimate residual readout, full OQC improves an AFBO baseline from 64.25 +/- 0.22 to 65.59 +/- 0.22, while OQC-LR reaches 65.52 +/- 0.25 with a substantially better speed-accuracy tradeoff. On TinyImageNet, the gated extension OQC-dynamic achieves 51.88 +/- 0.32, improving the baseline (50.45 +/- 0.21) by 1.43 points and outperforming all ungated variants. Mechanism analyses show near-zero post-projection auxiliary-main overlap together with improved representation geometry and class separation. The full family, including both ungated and gated variants, generalizes consistently across both datasets.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture evolution. Left: The host FFN (e.g., standard MLP or AFBO ) produces a dominant main branch. Middle: OQC adds a low-rank quadratic auxiliary feature and explicitly projects it onto the orthogonal complement of the main branch. Right: The gated OQC family maintains the orthogonal complement but dynamically or statically modulates its injection to the host branch.

Method	Acc. (%)	Params (M)	Img/s
AFBO + PR	64.25 $\pm$ 0.22	6.37	9762
OQC full	65.59 $\pm$ 0.22	6.97	5642
OQC-LR r56	65.52 $\pm$ 0.25	7.19	7497
OQC -static	65.44 $\pm$ 0.19	7.59	7032
OQC -dynamic	65.33 $\pm$ 0.11	7.59	7155

Table 1: Main CIFAR-100 results under the common Deep-ViT protocol. Full OQC is the strongest cross-validated variant, while OQC-LR r56 offers a better speed–accuracy trade-off.

Method	Acc. (%)	Params (M)	Img/s
AFBO + PR	50.45 $\pm$ 0.21	6.43	11900
OQC full	51.36 $\pm$ 0.28	7.03	8085
OQC-LR r56	51.35 $\pm$ 0.11	7.25	9400
OQC -static	51.47 $\pm$ 0.07	7.65	9108
OQC -dynamic	51.88 $\pm$ 0.32	7.65	9051

Table 2: TinyImageNet cross-dataset transfer (3 seeds, consistent protocol). All OQC variants improve the AFBO baseline. OQC -dynamic achieves the highest accuracy, confirming that the gated family generalizes beyond CIFAR-100.

Method	Acc.	EffRank	PartRatio	Sep.
AFBO + PR	64.41	76.64	48.49	1.0118
OQC full	65.18	78.17	50.43	1.0146
OQC-LR r56	65.29	77.53	49.85	1.0194
OQC -static	65.44	77.67	50.05	1.0172
OQC -dynamic	65.33	77.49	49.96	1.0202

Table 3: Mechanism statistics on CIFAR-100. “Sep.” denotes class separation score.

Figure 2: Mechanism visualization. (a) Auxiliary–main absolute cosine overlap before and after complement projection; orthogonalization reduces it to machine-precision zero. (b) Effective-rank vs. separation score: full OQC achieves the richest geometry while OQC -dynamic achieves the best class separation. (c) Complement usage: OQC -dynamic lowers average gate activation and introduces genuine input-dependent variation (high gate std.), confirming it is not a reparameterized static gate.

Figure 3: Accuracy–efficiency trade-off. (a) Rank sweep: full OQC improves monotonically with rank, while OQC-LR peaks at

r{=}56

(starred). (b) Pareto frontier: OQC-LR r56 delivers

65.52\%

7{,}485

img/s, substantially faster than full OQC (

5{,}655

img/s) while nearly matching its accuracy.

Condition	MLP	AFBO
Base	47.27 $\pm$ 0.42	62.82 $\pm$ 0.45
$+$ PR	47.29 $\pm$ 0.25	63.77 $\pm$ 0.11
$+$ OQC-LR	50.75 $\pm$ 0.23	63.88 $\pm$ 0.29
$+$ OQC-LR $+$ PR	51.39 $\pm$ 0.42	65.16 $\pm$ 0.07

Table 4: Component decomposition on CIFAR-100 (3 seeds, top-1 acc.%). Base : host FFN only, no readout, no complement. PR : penultimate-residual readout. OQC-LR : efficient low-rank complement.

Explore similar work

RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers
Oct 3, 2026Mengyuan Fan, Bokai Huang, JiaMing Pan +4Low-Bit Quantization Rotation-Based Quantization
Peking University
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training 4-Bit Quantization
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
Sep 28, 2026Peilin Sun, Guang Liang, Jin Tong +1Post-Training Quantization Low-Bit Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China

Oct 3, 2026J/K move · Enter open · S save

RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

Mengyuan Fan, Bokai Huang, JiaMing Pan, Xiaokun Yuan+3

Peking University

Low-Bit Quantization Rotation-Based Quantization Efficient ViTs Vision Transformer Quantization-Aware Training Image Classification

Open paper AskPDF arXiv

Vision Transformers (ViTs) achieve strong performance on image recognition and mobile vision applications, but their high-dimensional linear projections and attention computations still impose substantial storage and inference costs. Extremely low-bit quantization is a promising solution, yet ViTs often suffer severe accuracy degradation because conventional real-valued scalar codebooks are poorly matched to the directional geometry of Transformer projections. We present RPFQ-ViT, a Rotated Phase-Frame Quantization method that quantizes paired channels in two-dimensional phase planes, enabling low-bit codes to better preserve projection directions while recovering magnitude with lightweight scaling. RPFQ-ViT serves as a drop-in QAT replacement for nn.Linear and does not modify the standard real-valued attention, normalization, or activation computation graph. On ImageNet-1K, RPFQ-ViT-B/16 reaches 79.33% Top-1 / 94.48% Top-5 under W2/A4, Swin-T reaches 79.30% Top-1 / 94.79% Top-5 under W2/A8, and DeiT-S reaches 77.41% Top-1 / 93.11% Top-5 under W2/A8. Ablations, phase-geometry analysis, and direction-preservation metrics show that channel pairing, learnable rotation, phase-anchor learning, and residual phase refinement each improve quantization quality. We further deploy RPFQ-ViT image-classification models on native iOS and Android runtime stacks; with 2-bit packed weights, model size shrinks by roughly

5.4

7.1\times

relative to FP32 and end-to-end on-device latency drops by

1.4

1.6\times

. All ImageNet results trained in our codebase use a matched 300-epoch recipe and are reported as mean accuracies over three independent runs. These results show that RPFQ-ViT provides a favorable trade-off among accuracy, compression, and practical mobile deployment for extremely low-bit ViTs.

Abstract

Explore similar work

RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers