Hadamard transforms have become a key tool for stabilizing low-precision training, but existing methods apply them uniformly across tensors and computation paths. We show that this one-size-fits-all strategy is inherently limited: Hadamard smoothing reduces quantization error only when its direction is properly aligned with the operand's outlier structure. Through a systematic study of weights, activations, and gradients in LLM training, we identify three stable outlier patterns, Row-wise, Column-wise, and None, and show that each outlier pattern pair in matrix multiplication requires a distinct transform or outlier-handling strategy. We propose AdaHOP, Adaptive Hadamard transform with Outlier-Pattern-aware strategy, which applies Inner Hadamard Transform (IHT) when inner-dimension mixing properly suppresses the operands' outliers, and selectively applies Outlier Extraction (OE) that extracts dominant outlier rows or columns into a high-precision path when it does not. With fused, hardware-aware Triton kernels, AdaHOP enables training from scratch at MXFP4 precision with BF16-level quality, while achieving up to 3.6X memory compression, 1.46X end-to-end training speedup over BF16.
Figures & tables
Figure 1: Training loss curves and loss difference relative to BF16 for (Top) Llama3.2-1B and (Bottom) Instella-3B. AdaHOP consistently achieves the lowest loss gap relative to BF16 among all MXFP4-based methods.
Figure 2: (Left) 3D visualization of weight, activation, and gradient tensors from Llama3.2-1B’s block.2.self_attn_o . (Right) A column-wise outlier tensor after different Hadamard directions. Right Hadamard suppresses column-wise outliers; Left Hadamard leaves the structure intact.
Llama3.2-1B
Instella-3B
Llama3.1-8B
Pattern Pair
Total
Fwd
∇W
∇X
Total
Fwd
∇W
∇X
Total
Fwd
∇W
∇X
RN
23
0
15
8
79
0
33
46
65
0
27
38
RC
69
0
69
0
111
0
111
0
92
0
92
0
NN
35
15
0
20
100
36
2
62
103
32
4
67
NC
20
0
20
0
60
0
60
0
63
0
63
0
CN
181
97
0
84
361
216
1
144
312
192
1
119
Table 1: Distribution of outlier pattern pairs across computation paths for three LLM architectures after 30 training steps on C4. Patterns RR, NR, and CR do not appear because weights consistently exhibit None, and activations exhibit Column-wise or None.
Figure 3: IHT is effective only when its smoothing direction matches the outlier structure. We report the relative MSE reduction from IHT over all nine outlier-pattern pairs. Results are shown for synthetic tensors with five random seeds (Left) and real tensors from Llama3.2-3B’s block.11.feed_forward.w2 (Right). Improvement is defined as (Ebase−EIHT)/Ebase×100 ; see Appendix D for the full setup and error definitions.
Figure 4: Outlier patterns across 300 training steps for Llama3.2-3B. Patterns remain stable throughout training, enabling one-time calibration. Other layers in Appendix C .
Figure 5: AdaHOP pipeline. For each linear layer’s three matrix multiplications, AdaHOP selectively applies IHT, OE (Left or Right), or BF16 based on the detected outlier pattern pair Pn ( n=1,3,5 ).
Model
Tokens
Method
PIQA
HellaSwag
ARC-E
Lambada
Avg.
Llama3.2-1B
40B
BF16
72.69
52.33
47.85
38.17
52.76
MXFP4
64.69
37.94
37.29
11.93
37.96
MXFP4+Had.
71.85
51.37
45.91
37.68
51.70
Tseng et al.
72.14
50.95
46.17
36.81
51.52
HALO
72.09
51.89
46.72
39.03
52.43
AdaHOP-Lv1
73.04
52.24
46.91
38.21
52.60
Table 2: Zero-shot accuracy (%) after training on C4. Best MXFP4-based result is in bold ; second best is underlined .
Method
Mem.
Speed
Accel.
Quality
BF16
76.00
10930
1.00
✓
MXFP4
20.19
17640
1.61
✗
MXFP4+Had.
20.60
16305
1.49
✗
Tseng et al.
20.60
14881
1.36
▲
HALO
20.60
14251
1.30
▲
AdaHOP-Lv1
20.94
15804
1.45
✓
Table 3: (Left) End-to-end efficiency on Llama3.1-8B training with seq_len=4096 and BS=8 . Memory (GB), Speed (Token/sec), and acceleration ( × ) are measured for the full training pipeline. Quality: ✓ within 0.3pt of BF16, ▲ within 1pt, ✗ otherwise, based on the average downstream score across all models. (Right) kernel latency (ms) breakdown for two representative Llama3.1-8B GEMMs.
Figure 6: Throughput per batch size on Llama3.1-8B with seq_len=4096 . Highlighted number is the speedup of AdaHOP-Lv2. As the batch size increases, acceleration is maximized.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Per-column mean magnitude, variance, and quantization error of the activation tensor from Llama3.2-3B’s block.11.feed_forward.w2 . Red bars indicate the top-32 columns for each metric; light blue bars indicate the remaining columns.
Figure 8: Outlier patterns of weight ( W ), activation ( X ), and gradient ( GY ) tensors across 300 training steps for all 12 representative blocks of Llama3.2-3B. Each row represents a tensor from a specific layer, and the color indicates the detected outlier pattern at each step. This extended view reveals depth-dependent transitions in gradient outlier patterns: Row-wise patterns appear in early blocks, fade to None in the middle layers, and re-emerge selectively in the K-projection gradient toward the final blocks.
Figure 9: Hardware-aware implementation pipeline of AdaHOP. The pipeline consists of four stages: FOID, IHT and Quantization, Fused MXFP4+BF16 GEMM, and Fused Scatter-Add, all implemented as optimized Triton kernels targeting the AMD CDNA4 architecture. Note that the red block in Fast Outlier Index Detection represents the first 64 elements along each column.
Figure 10: MSE and latency as a function of the OE extraction rank for (Left) real tensors from Llama3.1-8B and (Right) synthetic tensors. Rank 64 achieves a favorable accuracy–efficiency trade-off in both settings: MSE is substantially reduced while latency remains close to the minimum.
Post-training quantization (PTQ) is essential for deploying LLMs under memory and bandwidth constraints. However, extreme low-bit quantization remains highly sensitive to activation outliers and anisotropic weight curvature. Existing incoherence-based PTQ methods mitigate this issue with fixed randomized Hadamard transforms (RHTs), which improve quantization robustness but cannot adapt the rotated basis to the layer, calibration distribution, or quantizer. We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided orthogonal processor that replaces fixed Hadamard mixing while preserving exact full-precision equivalence. HARP represents each rotation as a product of sparse butterfly-like block-orthogonal stages, supports non-power-of-two dimensions through Mixed-Radix schedules, and initializes to the RHT processor up to a fixed permutation. Fitted only on calibration data, HARP adapts the quantization basis to each layer and backend. Across 2--4-bit settings on Llama models from 1B to 70B, HARP consistently improves perplexity and yields its clearest zero-shot gains at 2 bits; a 2-bit Qwen3-8B experiment shows the same transfer beyond the Llama family. HARP also preserves deployment efficiency: on Llama 2 7B at 2 bits, it reaches 128 tok/s, retaining 90% of RHT throughput (142 tok/s) and running approximately 2.1× faster than FP16 (61 tok/s).
Artur Zagitov, Gleb Molodtsov, Aleksandr Beznosikov
Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions. Across LLaMA-style pretraining with 60M-1B models, 1x-8x Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.
Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1
California Institute of Technology · University of Copenhagen · Aarhus University
FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a seemingly complex outlier-suppression problem admits a truly simple solution: isolate the coherent mean before quantization. We therefore propose Averis, a mean-residual splitting quantization method that separates the mean component using only reductions and elementwise subtractions before FP4 quantization. Across Qwen3 0.6B Dense trained on 100B tokens and Qwen3 7B A1.5B MoE trained on 50B tokens, Averis enables robust W4A4G4 FP4 training, reducing BF16 loss gaps to 1.19%/0.81% versus 2.05%/1.10% for NVIDIA's recently released Hadamard-based outlier-smoothing method, while limiting downstream gaps to 0.89/0.71 points. With only 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA's Hadamard-based design, Averis provides a hardware-efficient path to stable low-bit LLM training. Complementary to Hadamard, Averis further reduces the Qwen3-0.6B loss and downstream gaps to 0.94% and 0.73 points when combined. Code is available at: https://anonymous.4open.science/r/averis-504D.
Hengjie Cao, Zhendong Huang, Mengyi Chen +15
Fudan University · Shanghai Innovation Institute · University of Bath +5