Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bottleneck for quantization stability due to its sensitivity to outliers and state-dependent Jacobian. We theoretically establish that suppressing the norm of this Jacobian helps in bounding quantization-induced performance degradation. Based on this, we propose Jacobian-Guided Noise Injection, a training strategy that injects zero-mean Gaussian noise into pre-attention logits, with variance derived directly from the Jacobian Frobenius norm. Unlike prior approaches that rely on heuristic or penalise jacobian directly, our method provides a way to identify the optimal noise variance based on the local attention sensitivity. We evaluate the method on SOTA LLM architectures, where it demonstrates improved robustness over popular PTQ methods. Empirical analysis reveals that the proposed method gives up to +37% relative gains on Top-1 accuracy on ImageNet-1K for SigLIP and improves relative perplexity by upto 40% on WikiText for language models in low bit quantisation settings, proving the efficacy of the approach.
Figures & tables
Figure 1: PTQ results on ImageNet-1K Top-1 accuracy for SigLIP base 16-384 with ERQ and RepQViT.
Figure 2: Activation cosine similarity for attention outputs in Siglip-base-384.
AWQ
GPTQ
SpinQuant
Model
Precision
Variant
PPL ↓
Acc ↑
PPL ↓
Acc ↑
PPL ↓
Acc ↑
Llama-3.2-3B
W4A4
Base
105.06
42.06
454.35
40.31
10.72
60.25
Ours
104.34
42.20
496.38
41.00
11.18
60.15
W4A8
Base
9.96
66.65
37.40
64.81
8.31
64.48
Ours
9.86
67.35
31.47
66.36
8.47
65.17
W6A6
Base
10.80
64.07
10.87
63.69
8.03
65.98
Table 1: Zero shot PTQ results for Llama-3.2-3B and Qwen2.5-3B. Acc is average accuracy of over 7 benchmarks ( ↑ ).
Figure 3: (a) Jacobian norm during training for Qwen2.5-3B layers 9, 18, and 27. (b) Effect of scaling coefficient α on SigLIP W4A4 QAT (ImageNet-1K zero-shot accuracy). Moderate scaling ( α∈[0.5,1.0] ) yields better results compared to no noise ( α=0 ).
Figure 4: Attention maps for SigLIP (layer 10). Noise-trained models exhibit more diffused attention patterns across key positions.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Weight Bits
Activation Bits
W4A4
4
4
W4A8
4
8
W5A5
5
5
W6A6
6
6
W6A8
6
8
W8A8
8
8
Appendix
Table 6
Method
Noise
W4A4
W5A5
W6A6
W8A8
ERQ
B
51.2
72.4
81.3
85.9
B+N
60.1
76.8
83.5
86.4
RepQViT
B
32.5
65.1
78.9
85.6
B+N
69.8
74.2
82.1
86.2
Appendix
Table 2: SigLIP base 16-384 PTQ results on ImageNet-1K zero-shot classification. All values are Top-1 accuracy (%). B: Baseline, B+N: Baseline with Jacobian-guided Noise. The FP32 baseline achieves 86.7% Top-1 accuracy.
Model
MMLU
HellaSwag
PIQA
Winogrande
TruthfulQA
WikiText PPL
Qwen2.5-3B (B)
65.25
75.08
79.87
70.72
47.41
10.60
Qwen2.5-3B (B+N)
65.48
74.91
79.54
70.17
47.47
10.46
Appendix
Table 3: Qwen2.5-3B full-precision (FP16) results with and without noise injection. B: Baseline, B+N: With Jacobian Noise ( α=0.5 , rowwise).
Model
Variant
Precision
PPL
HellaSwag
PIQA
WinoGrande
ARC-E
ARC-C
BoolQ
TQA-MC2
Avg7
Qwen2.5-3B
Base
W4A4KV4
10.49
66.27
74.32
59.19
71.93
44.20
65.32
41.12
60.34
W4A8KV8
8.55
72.91
78.02
68.98
77.48
48.21
79.94
45.91
67.35
W6A6KV6
8.36
73.92
78.35
67.25
77.15
48.38
79.08
46.64
67.25
W6A8KV8
8.32
74.25
79.33
68.59
76.18
48.98
79.57
46.74
67.66
Jac-noise
W4A4KV4
10.33
66.65
74.43
61.88
73.70
45.05
70.18
46.42
62.62
W4A8KV8
8.35
72.32
78.40
68.35
74.03
45.73
77.55
43.25
65.66
Appendix
Table 4: SpinQuant W4A4KV4 / W4A8KV8 / W6A6KV6 / W6A8KV8 full results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2.
Model
Variant
Precision
PPL
HellaSwag
PIQA
WinoGrande
ARC-E
ARC-C
BoolQ
TQA-MC2
Avg7
Qwen2.5-3B
Base
W4A4
8121.24
30.23
54.35
48.54
32.32
23.12
48.07
47.45
40.58
W4A8
10.80
74.23
77.97
68.11
78.32
49.57
80.18
46.21
67.80
W6A6
15.29
70.26
74.86
64.72
75.80
49.40
75.54
45.25
65.12
W8A8
10.80
74.23
77.80
68.59
77.99
50.00
80.70
45.57
67.84
Jac-noise
W4A4
3281.04
30.57
53.26
52.25
33.54
24.06
49.42
47.80
41.56
W4A8
11.01
73.45
78.24
68.51
78.11
49.40
80.03
44.61
67.48
Appendix
Table 5: AWQ W4A4 / W4A8 / W6A6 / W8A8 results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2.
Model
Variant
Precision
PPL
HellaSwag
PIQA
WinoGrande
ARC-E
ARC-C
BoolQ
TQA-MC2
Avg7
Qwen2.5-3B
Base
W4A4
5172.39
25.81
49.95
51.14
26.52
23.46
43.91
49.27
38.58
W4A8
14.13
72.05
77.15
65.98
78.00
48.89
80.03
44.43
66.65
W6A6
28.42
63.32
70.78
60.46
69.15
42.66
68.23
45.19
59.97
W8A8
10.98
73.99
78.02
67.88
77.99
49.15
80.06
45.78
67.55
Jac-noise
W4A4
4207.85
26.31
50.05
51.04
26.00
24.32
45.38
49.64
38.96
W4A8
12.21
71.39
77.42
66.77
77.23
49.40
77.43
43.14
66.11
Appendix
Table 6: GPTQ W4A4 / W4A8 / W6A6 / W8A8 results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2.
Figure 5: Layer-wise cosine similarity between quantized (15-bit) and unquantized activations for a VLM deployed on edge hardware. (a) Baseline model shows severe similarity degradation, with some layers dropping to near-zero cosine similarity. (b) Model trained with Jacobian-guided noise injection maintains consistently high similarity ( > 0.95) across all layers.
Figure 6: Attention maps for SigLIP (layer 10) Baseline vs baseline with noise. Noise-trained models exhibit more diffused attention patterns across key positions.