SignRot: LLM Quantization with Massive Outlier-Aware Sign-Adjusted Rotation
Organizations: Hanyang University · University of California, Santa Barbara
Abstract
While quantization compresses LLMs and accelerates inference, outlier activations impede efficient low-bit representation. In particular, massive outliers cause substantial performance degradation. Although rotation-based methods that rely on randomized Hadamard transforms for online inference aim to mitigate this, they can produce fragmented distributions with empty bins when a massive outlier dominates a token's activation. This fragmentation severely undermines uniform quantization performance. Moreover, massive outliers concentrate in the down-projection input, where the preceding nonlinearity requires the rotation to be applied at inference time. This constraint prevents training-based approaches such as SpinQuant from learning the online rotation at this position, as a learned dense matrix cannot use the Fast Walsh-Hadamard Transform and would require a full matrix multiplication during inference. To address this, we propose SignRot (Sign-adjusted Rotation), which identifies massive outlier channels and constructs a sign adjustment vector that unifies the signs of the corresponding Hadamard row, collapsing the bimodal distribution into a unimodal profile amenable to quantization. Since the aligned outlier shifts the activation mean away from zero, we pair SignRot with asymmetric activation quantization, whose zero-point absorbs this shift. As a lightweight, drop-in improvement compatible with any Hadamard-based quantization method, SignRot requires only a single calibration forward pass and adds only element-wise sign flips at inference, without an additional matrix multiplication. Applying SignRot to QuaRot achieves a WikiText2 perplexity of 5.88 and an average zero-shot reasoning accuracy of 72.73, surpassing SpinQuant on the LLaMA3-70B model. Code is available at https://github.com/HYU-ARK-Lab/SignRot.
Figures & tables
| Method | W Quantizer | 2-7B | 2-13B | 2-70B | 3-8B | 3-70B | ||||||||||
| Wiki2 | C4 | Avg. | Wiki2 | C4 | Avg. | Wiki2 | C4 | Avg. | Wiki2 | C4 | Avg. | Wiki2 | C4 | Avg. | ||
| PPL | PPL | 0-shot | PPL | PPL | 0-shot | PPL | PPL | 0-shot | PPL | PPL | 0-shot | PPL | PPL | 0-shot | ||
| FP16 | - | 5.47 | 7.26 | 69.79 | 4.88 | 6.73 | 72.55 | 3.32 | 5.71 | 77.05 | 6.14 | 9.45 | 73.23 | 2.86 | 7.17 | 79.95 |
| QuaRot | RTN | 8.56 | 11.86 | 57.73 | 6.10 | 8.67 | 66.25 | 4.14 | 6.42 | 73.47 | 10.60 | 17.19 | 61.34 | 55.44 | 79.48 | 35.36 |
| QuaRot+SignRot | RTN | 7.93 | 11.21 | 60.43 | 6.00 | 8.48 | 67.51 | 4.05 | 6.32 | 73.86 | 9.68 | 15.72 | 64.17 | 41.47 | 58.26 | 38.52 |
| QuaRot | GPTQ | 6.10 | 8.32 | 65.01 | 5.40 | 7.54 | 68.91 | 3.79 | 6.12 | 75.68 | 8.16 | 13.38 | 65.79 | 6.60 | 12.87 | 70.45 |
| Model | Method | W Quantizer | Wiki2 | C4 | ARC-C | ARC-E | HellaSwag | LAMBADA | PIQA | Winogrande | Avg. |
| PPL | PPL | 0-shot | 0-shot | 0-shot | 0-shot | 0-shot | 0-shot | 0-shot | |||
| Mistral-7B-v0.3 | FP16 | - | 5.32 | 8.47 | 52.30 | 78.20 | 80.41 | 75.32 | 82.26 | 73.88 | 73.73 |
| QuaRot | GPTQ | 5.80 | 9.19 | 48.89 | 76.13 | 78.09 | 73.18 | 79.86 | 72.37 | 71.42 | |
| QuaRot+SignRot | GPTQ | 5.71 | 9.09 | 48.72 | 76.09 | 78.95 | 72.64 | 80.30 | 72.45 | 71.53 | |
| Qwen3-30B-A3B | FP16 | - | 8.70 | 14.05 | 56.57 | 79.25 | 77.63 | 64.76 | 80.47 | 69.38 | 71.34 |
| QuaRot | RTN | 11.70 | 18.21 | 47.18 | 68.90 | 69.00 | 52.73 | 75.57 | 63.38 | 62.79 |
| Method | Prefill | Decode | E2E | Peak memory | ||||
| ms | Speedup | ms/token | Speedup | ms | Speedup | GB | Saving | |
| FP16 | 496.7 | 1.00 | 24.01 | 1.00 | 3626.6 | 1.00 | 13.84 | 1.00 |
| QuaRot | 283.4 | 1.75 | 14.92 | 1.61 | 2194.0 | 1.65 | 4.33 | 3.19 |
| QuaRot+SignRot | 287.3 | 1.73 | 14.97 | 1.60 | 2203.5 | 1.65 | 4.33 | 3.19 |
| Overhead of SignRot | +1.37% | +0.34% | +0.43% | +0.02% | ||||
| Model | Method | Act asym | PPL ( ) | Acc ( ) | |||||||
| Wiki2 | C4 | ARC-C | ARC-E | HellaSwag | LAMBADA | PIQA | Winogrande | Avg. | |||
| 2-7B | QuaRot | ✗ | 6.12 | 8.31 | 42.41 | 70.26 | 72.62 | 70.61 | 76.99 | 66.35 | 66.54 |
| QuaRot+SignRot | ✗ | 6.12 | 8.32 | 41.73 | 69.48 | 72.37 | 70.50 | 77.15 | 66.74 | 66.33 | |
| QuaRot | ✓ | 6.06 | 8.23 | 42.42 | 70.16 | 72.70 | 70.93 | 77.34 | 66.13 | 66.62 | |
| QuaRot+SignRot | ✓ | 6.05 | 8.23 | 42.08 | 69.89 | 72.70 | 71.13 | 77.20 | 66.91 | 66.65 | |
| 3-8B | QuaRot | ✗ | 8.15 | 13.28 | 44.71 | 70.62 | 73.66 | 68.18 | 75.56 | 67.29 | 66.67 |
| Calibration Dataset | Wiki2 PPL | C4 PPL | Avg. Acc |
| Wiki2 (Seed 0) | 7.85 | 12.74 | 66.66 |
| Wiki2 (Seed 1) | 7.87 | 12.80 | 67.04 |
| Wiki2 (Seed 2) | 7.84 | 12.77 | 67.41 |
| C4 (Seed 0) | 7.88 | 12.78 | 67.26 |
| C4 (Seed 1) | 7.82 | 12.77 | 66.52 |
| C4 (Seed 2) | 7.87 | 12.82 | 67.25 |
| Adjusted input | PPL ( ) | Avg. Acc ( ) | |
| Wiki2 | C4 | 0-shot | |
| 7.957 0.039 | 12.955 0.060 | 67.21 0.39 | |
| 7.972 0.050 | 12.964 0.073 | 67.10 0.59 | |
| 7.974 0.035 | 12.988 0.052 | 66.94 0.69 | |
| 7.898 0.019 | 12.869 0.029 | 67.74 0.63 | |
| + | 7.941 0.034 | 12.933 0.035 | 67.46 0.38 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Massive Outlier Token | Non-outlier Token | |||||||
| Mean | Variance | Min | Max | Mean | Variance | Min | Max | |
| Original | 0.1377 | 207.3750 | -46.1562 | 1510.0000 | -0.0000 | 0.0001 | -0.2783 | 0.5435 |
| After RHT | 0.0001 | 207.3750 | -15.6328 | 15.6250 | 0.0000 | 0.0001 | -0.0419 | 0.0378 |
| With SignRot | 14.3906 | 0.3059 | 13.1328 | 15.6328 | 0.0000 | 0.0001 | -0.0419 | 0.0370 |
| Tokens/seq | Channels/token | ||||||
| Model | Layer | Sun et al. [ 30 ] | Ours | Sun et al. [ 30 ] | Ours | energy (%) | |
| LLaMA2-7B | 2 | 2.09 | 0.97 | 1.52 | 2.44 | 0.03 | 99.8 |
| 31 | 2.09 | 0.98 | 2.00 | 4.98 | 0.53 | 78.1 | |
| LLaMA2-13B | 4 | 1.09 | 1.09 | 1.00 | 3.00 | 0.08 | 99.0 |
| 39 | 1.09 | 1.09 | 1.00 | 1.00 | 0.05 | 99.3 | |
| 40 | 0.23 | 0.05 | 1.20 | 1.00 | 0.34 | 61.2 | |
| Tokens/seq | Channels/token | ||||||
| Model | Layer | Sun et al. [ 30 ] | Ours | Sun et al. [ 30 ] | Ours | energy (%) | |
| LLaMA2-70B | 3 | 2.05 | 1.09 | 1.00 | 2.04 | 0.13 | 98.4 |
| 9 | 1.95 | 0.86 | 3.05 | 3.98 | 0.49 | 79.1 | |
| 13 | 0.95 | 0.99 | 1.00 | 1.00 | 0.01 | 99.9 | |
| 60 | 0.01 | 0.05 | 1.00 | 1.43 | 0.29 | 87.5 | |
| 62 | 0.01 | 0.12 | 1.00 | 1.00 | 0.07 | 86.6 | |
| Model | Layer | Rank | QuaRot | QuaRot+SignRot | all (%) | others (%) |
| LLaMA2-7B | 2 | 3 | 0.0055 | 0.0021 | ||
| 31 | 1 | 0.0072 | 0.0073 | |||
| LLaMA2-13B | 4 | 1 | 0.0052 | 0.0033 | ||
| 39 | 3 | 0.0031 | 0.0031 | |||
| 40 | 6 | 0.0025 | 0.0025 | |||
| LLaMA3-8B | 2 | 1 | 0.0255 | 0.0098 |
| Model | Layer | Tokens/seq | Top channels | (%) | Range ratio |
| LLaMA2-7B | 2 / 32 | 0.97 | 1 | 100.0 | 11.34 |
| LLaMA2-13B | 4 / 40 | 1.09 | 1 | 100.0 | 7.00 |
| LLaMA2-70B | 3 / 80 | 1.09 | 1 | 100.0 | 6.05 |
| 9 / 80 | 0.87 | 1 | 99.1 | 2.26 | |
| 13 / 80 | 0.99 | 1 | 100.0 | 9.40 | |
| LLaMA3-8B | 2 / 32 | 1.00 | 1 | 100.0 | 7.47 |
| Method | Rotation | Act. quantization | Weight granularity | GPTQ damping |
| QuaRot | Randomized Hadamard | Per-token symmetric | Per-channel | 0.01 |
| QuaRot+SignRot | Randomized Hadamard | Per-token asymmetric | Per-channel | 0.01 |
| SpinQuant | Released learned rotation | Per-token asymmetric | Per-channel | 0.01 |
| SpinQuant+SignRot | Released learned rotation | Per-token asymmetric | Per-channel | 0.01 |
| DartQuant | Trained rotation | Per-token asymmetric | Group 128 | 0.1 |
| DartQuant+SignRot | Trained rotation | Per-token asymmetric | Group 128 | 0.1 |
| Method | Act. clip | K clip | V clip |
| QuaRot+SignRot | 0.9 a | 0.9 | 0.9 |
| SpinQuant+SignRot | 0.9 | 0.9 | 0.9 |
| DartQuant+SignRot | 0.9 | 1.0 | 1.0 |
| Model | Method | W Quantizer | Wiki2 | C4 | ARC-C | ARC-E | HellaSwag | LAMBADA | PIQA | Winogrande | Avg |
| 2-7B | FP16 | - | 5.47 | 7.26 | 46.16 | 74.54 | 75.98 | 73.92 | 79.05 | 69.06 | 69.79 |
| QuaRot | RTN | 8.56 | 11.86 | 36.60 | 61.41 | 65.07 | 48.06 | 72.20 | 63.06 | 57.73 | |
| QuaRot+SignRot | RTN | 7.93 | 11.21 | 38.31 | 62.84 | 66.60 | 58.63 | 73.61 | 62.59 | 60.43 | |
| QuaRot | GPTQ | 6.10 | 8.32 | 42.32 | 68.35 | 72.53 | 65.40 | 76.33 | 65.11 | 65.01 | |
| QuaRot+SignRot | GPTQ | 6.09 | 8.31 | 42.83 | 70.08 | 73.26 | 71.73 | 78.07 | 67.17 | 67.19 | |
| 2-13B | FP16 | - | 4.88 | 6.73 | 49.15 | 77.44 | 79.39 | 76.73 | 80.47 | 72.14 | 72.55 |
| Model | Method | W Quantizer | Wiki2 | C4 | ARC-C | ARC-E | HellaSwag | LAMBADA | PIQA | Winogrande | Avg |
| 2-7B | FP16 | - | 5.47 | 7.26 | 46.16 | 74.54 | 75.98 | 73.92 | 79.05 | 69.06 | 69.79 |
| SpinQuant | RTN | 7.71 | 11.14 | 36.77 | 64.27 | 66.53 | 60.31 | 74.43 | 61.72 | 60.67 | |
| SpinQuant+SignRot | RTN | 7.69 | 10.94 | 37.97 | 64.48 | 65.88 | 60.88 | 74.76 | 62.04 | 61.00 | |
| SpinQuant | GPTQ | 6.08 | 8.25 | 41.21 | 68.94 | 72.95 | 71.63 | 77.64 | 65.27 | 66.27 | |
| SpinQuant+SignRot | GPTQ | 6.09 | 8.27 | 42.66 | 70.24 | 72.18 | 72.39 | 77.31 | 67.64 | 67.07 | |
| 2-13B | FP16 | - | 4.88 | 6.73 | 49.15 | 77.44 | 79.39 | 76.73 | 80.47 | 72.14 | 72.55 |
| Model | Method | Wiki2 | C4 | ARC-C | ARC-E | HellaSwag | LAMBADA | PIQA | Winogrande | Avg |
| 2-7B | DartQuant | 5.97 | 8.07 | 42.58 | 69.74 | 73.49 | 71.36 | 77.69 | 67.01 | 66.98 |
| DartQuant+SignRot | 5.94 | 8.04 | 42.58 | 70.58 | 73.83 | 71.92 | 77.48 | 67.25 | 67.27 | |
| 2-13B | DartQuant | 5.32 | 7.39 | 46.59 | 75.63 | 77.26 | 75.02 | 78.51 | 69.46 | 70.41 |
| DartQuant+SignRot | 5.31 | 7.37 | 47.35 | 75.34 | 76.87 | 74.64 | 79.33 | 70.01 | 70.59 | |
| 2-70B | DartQuant | 3.66 | 6.01 | 54.27 | 78.20 | 83.06 | 79.16 | 82.21 | 76.80 | 75.61 |
| DartQuant+SignRot | 3.66 | 6.00 | 54.86 | 79.00 | 82.90 | 78.92 | 82.10 | 77.35 | 75.86 |
| Methods | LLaMA2-7B | LLaMA2-13B | LLaMA2-70B |
| RTN, QuaRot+SignRot | 1+2min | 2+3min | 5+8min |
| GPTQ | 14min | 26min | 1.8h |
| GPTQ, QuaRot | 1+14min | 2+26min | 0.1h+1.8h |
| GPTQ, QuaRot+SignRot | 1+14min | 2+27min | 0.1h+1.8h |
| GPTQ, SpinQuant | 1.3h+0.2h | 2.4h+0.4h | 12h+1.8h |
| Model | Method | Act asym | Prefill (ms) | Decode (ms/token) | E2E (ms) | Peak memory (GB) |
| 2-7B | FP16 | - | 496.7 | 24.01 | 3626.6 | 13.84 |
| QuaRot | ✗ | 228.3 | 14.13 | 2037.6 | 4.24 | |
| QuaRot+SignRot | ✗ | 232.9 | 14.17 | 2047.3 | 4.24 | |
| QuaRot | ✓ | 283.4 | 14.92 | 2194.0 | 4.33 | |
| QuaRot+SignRot | ✓ | 287.3 | 14.97 | 2203.5 | 4.33 | |
| 2-13B | FP16 | - | OOM | |||
| Layer Size | Batch | FP16 | Int4 | +QuaRot | +SignRot | Speedup |
| 11008x4096 | 1 | 2.59 | 0.73 | 0.77 | 0.85 | 3.05 |
| 2 | 5.06 | 1.45 | 1.52 | 1.65 | 3.07 | |
| 4 | 9.85 | 2.92 | 3.04 | 3.29 | 2.99 | |
| 8 | 19.78 | 5.73 | 5.97 | 6.44 | 3.07 | |
| 16 | 39.71 | 11.27 | 11.79 | 12.76 | 3.11 | |
| 32 | 79.58 | 22.53 | 23.51 | 25.39 | 3.13 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers