S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations

Arnav Chavan; Nahush Lele; Udbhav Bamba; Sankalp Dayal; Aditi Raghunathan; Deepak Gupta

cs.LGFeb 16, 2026

S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations

Authors: Arnav Chavan, Nahush Lele, Udbhav Bamba, Sankalp Dayal, Aditi Raghunathan, Deepak Gupta

Organizations: Amazon · Carnegie Mellon University

Abstract

Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay ( $S^2D$ ), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that $S^2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly. Models trained with $S^2D$ achieve up to 7% improved PTQ accuracy on ImageNet under W4A4 quantization and 4% gains when combined with QAT. These improvements also generalize across downstream tasks and vision-language models, enabling the scaling of increasingly large and rigorously trained models without sacrificing deployment efficiency.

Figures & tables

Figure 1: Activation Outlier Suppression. Comparison of the absolute maximum activation value (Max Abs.) of the Layer-9-FC1 output in the SigLIP-2 Base model. AdamW and Muon produce large activation outliers, whereas

S^{2}D

substantially suppresses them, leading to improved downstream quantization performance as shown for W4A4.

Panel of Figure 2 — Figure 2: Activation outlier severity escalates with pre-training scale. Activation outlier severity escalates with increasing pre-training scale. The figure plots activation distributions from the feed-forward network (FFN) layers of ViT backbones across CLIP , SigLIP , and SigLIP2 . A clear upward trend emerges: the magnitude of activation outliers consistently increase as we move from CLIP $\rightarrow$ SigLIP $\rightarrow$ SigLIP2 , highlighting the heavier-tailed activation behavior induced by larger and more recent model families.

Model	Layer	PCDR 1	PCDR 3	$\sigma_{max}$
CLIP	Layer 1	0.08	0.09	3.78
	Layer 5	0.05	0.14	3.06
	Layer 9	0.18	0.19	4.81
SigLIP	Layer 1	0.01	0.02	3.40
	Layer 5	0.03	0.89	4.86
	Layer 9	0.14	0.14	7.59

Table 1: Principal Component Dominance Ratios for Outlier Activations. Top-1 and Top-3 PCDR for FFN layer activations, along with the maximum singular value of the corresponding FFN weights across CLIP, SigLIP and SigLIP2.

\sigma_{max}

denotes largest singular value.

Res.	Method	FP16	RepQ-ViT				PTQ4ViT				ERQ
Res.	Method	FP16	W4A4	W5A5	W6A6	W8A8	W4A4	W5A5	W6A6	W8A8	W4A4	W5A5	W6A6	W8A8
384	AdamW	85.0	37.4	46.0	58.5	83.0	3.4	44.4	65.3	83.4	65.6	78.5	81.1	83.8
384	Ours	85.0	41.4	78.0	80.0	83.5	3.9	62.0	81.5	84.1	73.0	81.9	82.7	84.0
512	AdamW	85.3	16.3	43.3	53.9	83.3	4.2	46.9	80.1	84.5	58.6	77.6	80.1	78.6
512	Ours	85.4	17.2	79.2	82.7	84.3	3.8	77.0	83.0	84.9	68.1	82.3	83.5	84.1

Table 2: SigLIP2 quantization performance on ImageNet1k . Comparing ImageNet1k performance on AdamW and AdamW+

S^{2}D

(Ours) across various weight (W) / activation (A) precision settings and post-training quantization (PTQ) methods. The table demonstrates that

S^{2}D

shows consistent improvements for W4A4, W5A5, W6A6, and W8A8 configurations, when using ERQ, PTQ4ViT, and RepQ-ViT to perform PTQ.

Metric	Optimizer	Layer 5	Layer 9
PCDR 1	AdamW	0.91	0.77
PCDR 1	Ours	0.46	0.09
Max Abs.	AdamW	176.4	1166.2
Max Abs.	Ours	59.7	614.7
$\sigma_{\max}$	AdamW	10.4	7.9
$\sigma_{\max}$	Ours	3.2	3.9

Table 3: Improved Layer Conditioning. PCDR 1 , maximum absolute activation, and maximum singular value for the FC1 layers of SigLIP2 after fine-tuning with AdamW and AdamW+

S^{2}D

(Ours). The results show that

S^{2}D

consistently reduces PCDR 1 compared to AdamW, indicating better spectral concentration. We also observe notable decreases in absolute max activation (Max Abs.) and leading singular values.

Figure 3: QAT Performance Gains. Quantization-Aware Training (QAT) results for W3A4 and W4A4 on ImageNet1K using AdamW and AdamW+

S^{2}D

(Ours). The bar plot shows that pairing QAT with

S^{2}D

improves downstream accuracy over vanilla QAT..

Task	Method	FP16	ERQ				PTQ4ViT
Task	Method	FP16	W4A4	W5A5	W6A6	W8A8	W6A6	W8A8
Object Detection	AdamW	50.1	1.8	10.8	45.8	48.2	2.2	42.1
Object Detection	Ours	50.3	16.6	40.7	47.8	49.7	24.7	48.2
Instance Segmentation	AdamW	43.6	0.50	11.7	39.9	41.9	1.9	36.5
Instance Segmentation	Ours	43.8	13.0	34.4	41.4	43.2	20.7	42.8

Table 4: Downstream Task Improvements. Quantization results for object detection and instance segmentation on MS-COCO. The table reports ERQ (W4A4, W5A5, W6A6, W8A8) and PTQ4ViT (W6A6, W8A8) performance, showing that AdamW+

S^{2}D

(Ours) delivers consistent performance gains across PTQ settings.

Benchmark	Method	FP16	W4A4	W5A5	W6A6
GQA	AdamW	47.6	35.3	38.5	43.9
GQA	Ours	55.4	40.1	47.6	52.8
TextVQA	AdamW	31.8	6.79	10.4	19.2
TextVQA	Ours	36.7	9.8	17.6	28.0
POPE	AdamW	69.8	66.6	67.4	69.8
POPE	Ours	70.3	66.6	66.6	69.4

Table 5: VLM Quantization. Evaluation of LLaVA-1.5 (SigLIP2-Base-384 + Qwen2.5-0.5B) under AdamW and AdamW+

S^{2}D

(Ours) fine-tuning.

S^{2}D

provides gains in both full-precision and quantized settings across GQA, TextVQA, POPE, and DocVQA, demonstrating its effectiveness as a spectral conditioning method for multimodal VLM pipelines.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	No PCDR	S 2 D
W4A4	40.1	73.0
W5A5	77.1	81.9
W6A6	81.2	82.7

Appendix

Table 6: Impact of PCDR-guided layer selection. Comparison of S 2 D with and without PCDR targeting on ImageNet classification using ERQ quantization (SigLIP2-Base-384). Without PCDR, uniform spectral regularization severely degrades low-bit performance.

		PCDR 1		Max Abs. Activation		Max Singular Value
Model	Layer	AdamW	S 2 D	AdamW	S 2 D	AdamW	S 2 D
SigLIP2	Layer 5	0.91	0.46	176.40	59.68	10.38	3.22
SigLIP2	Layer 9	0.77	0.09	1166.2	614.7	7.87	3.96
DINOv3	Layer 2	0.47	0.11	1048.41	440.38	37.29	17.94
DINOv3	Layer 4	0.45	0.22	115.55	67.56	8.67	3.79

Appendix

Table 7: Comparison of FFN activation and weight statistics for SigLIP2 and DINOv3. We report the PCDR 1 and maximum absolute activation of the FFN layers, along with the maximum singular value of their corresponding weights, after fine-tuning with AdamW and AdamW+S 2 D.

Method	W8A8	W6A6	W5A5
AdamW	58.03	57.04	28.52
AdamW + S 2 D	76.53	56.10	30.69

Appendix

Table 8: PTQ4ViT quantization results on DINOv3-Base with and without S 2 D regularization. ImageNet top-1 accuracy (%) is reported.

	S 2 D	Baseline
FP	28.2	29.2
W8A8	26.2	24.0
W7A7	24.7	22.1
W6A6	19.8	17.8

Appendix

Table 9: GSM8K 0-shot results with RTN quantization for Qwen2.5-0.5B. S 2 D improves quantized accuracy across all tested bit-widths using the same hyperparameters from the vision experiments.

Vary SVD update frequency $k$
$\boldsymbol{k}$	topk	$\boldsymbol{\tau}$	FP16 (%)	W4A4 (%)
10	3	0.95	85.0	72.2
100	3	0.95	85.0	73.0
Vary number of targeted singular values
100	5	0.95	85.0	72.5
100	10	0.95	84.9	72.6

Appendix

Table 10: Hyperparameter sensitivity analysis for S 2 D on ImageNet. We vary the SVD computation frequency (

k

), number of targeted singular values ( topk ), and PCDR threshold. The default configuration is shown in bold.

	Baseline	$\boldsymbol{n{=}2}$	$n{=}3$	$n{=}4$
W4A4	65.6	73.0	68.3	69.6
W5A5	78.5	81.9	79.0	78.7
W6A6	81.1	82.7	81.7	82.4

Appendix

Table 11: Sensitivity to power exponent

n

. ImageNet accuracy (%) using ERQ quantization on SigLIP2-Base-384.

n{=}2

provides the best balance across bit-widths.

Activation Outliers

Activation outliers are unusually large or structured neural-network activations that complicate quantization, training stability, and efficient inference.

See all 17 papers

Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
May 2, 2026Jin Tong, Guang Liang, Peilin Sun +1Vision Transformer Activation Outliers
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs
Mar 4, 2026Pranav Kumar KaliaperumalActivation Outliers Post-Training Quantization
M.S. Computer Science, University of Colorado Denver, Aurora, Colorado, USA
Activation Sensitivity as a Unifying Principle for Post-Training Quantization
Jan 15, 2026Bruce Changlong XuLLM Quantization Post-Training Quantization

May 2, 2026J/K move · Enter open · S save

Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay

Jin Tong, Guang Liang, Peilin Sun, Jianxin Wu

State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China

Vision Transformer Activation Outliers Activation Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Low-bit quantization is a practical route for efficiently deploying vision Transformers, yet activation outliers complicate fully quantized deployment. Existing methods either handle quantization post-training or suppress large activations during training; however, aggressively restricting outliers in vision models can lead to a poorer trade-off between full-precision and quantized accuracy. We argue that rather than simply suppressing outliers, the training objective should control the structural amplification that makes them harmful. To this end, we introduce Colinearity-Decay (CD), a structural regularizer for ordered matrix pairs within Transformer blocks. CD penalizes detrimental cross-matrix alignment and mitigates extreme activations without altering the architecture or task loss. Applied as a decoupled update, CD is non-invasive and introduces minimal training overhead. Across ImageNet-1K pre-training, COCO detection, and downstream fine-tuning, CD consistently boosts quantized accuracy across multiple pipelines while preserving, or even improving, full-precision performance. Ultimately, our results demonstrate that structural regularization effectively prepares vision Transformers for low-bit deployment with zero inference-time overhead.

Abstract

Activation Outliers

Explore similar work

Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay

Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

Activation Sensitivity as a Unifying Principle for Post-Training Quantization