AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

Seonggon Kim; Alireza Khodamoradi; Pranathi Vasireddy; Kristof Denolf; Eunhyeok Park

cs.LGApr 2, 2026

AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

Authors: Seonggon Kim, Alireza Khodamoradi, Pranathi Vasireddy, Kristof Denolf, Eunhyeok Park

Organizations: Advanced Micro Devices, Inc. · POSTECH

Abstract

Hadamard transforms have become a key tool for stabilizing low-precision training, but existing methods apply them uniformly across tensors and computation paths. We show that this one-size-fits-all strategy is inherently limited: Hadamard smoothing reduces quantization error only when its direction is properly aligned with the operand's outlier structure. Through a systematic study of weights, activations, and gradients in LLM training, we identify three stable outlier patterns, Row-wise, Column-wise, and None, and show that each outlier pattern pair in matrix multiplication requires a distinct transform or outlier-handling strategy. We propose AdaHOP, Adaptive Hadamard transform with Outlier-Pattern-aware strategy, which applies Inner Hadamard Transform (IHT) when inner-dimension mixing properly suppresses the operands' outliers, and selectively applies Outlier Extraction (OE) that extracts dominant outlier rows or columns into a high-precision path when it does not. With fused, hardware-aware Triton kernels, AdaHOP enables training from scratch at MXFP4 precision with BF16-level quality, while achieving up to 3.6X memory compression, 1.46X end-to-end training speedup over BF16.

Figures & tables

Figure 1: Training loss curves and loss difference relative to BF16 for (Top) Llama3.2-1B and (Bottom) Instella-3B. AdaHOP consistently achieves the lowest loss gap relative to BF16 among all MXFP4-based methods.

Figure 2: (Left) 3D visualization of weight, activation, and gradient tensors from Llama3.2-1B’s block.2.self_attn_o . (Right) A column-wise outlier tensor after different Hadamard directions. Right Hadamard suppresses column-wise outliers; Left Hadamard leaves the structure intact.

	Llama3.2-1B				Instella-3B				Llama3.1-8B
Pattern Pair	Total	Fwd	$\nabla W$	$\nabla X$	Total	Fwd	$\nabla W$	$\nabla X$	Total	Fwd	$\nabla W$	$\nabla X$
RN	23	0	15	8	79	0	33	46	65	0	27	38
RC	69	0	69	0	111	0	111	0	92	0	92	0
NN	35	15	0	20	100	36	2	62	103	32	4	67
NC	20	0	20	0	60	0	60	0	63	0	63	0
CN	181	97	0	84	361	216	1	144	312	192	1	119

Table 1: Distribution of outlier pattern pairs across computation paths for three LLM architectures after 30 training steps on C4. Patterns RR, NR, and CR do not appear because weights consistently exhibit None, and activations exhibit Column-wise or None.

Figure 3: IHT is effective only when its smoothing direction matches the outlier structure. We report the relative MSE reduction from IHT over all nine outlier-pattern pairs. Results are shown for synthetic tensors with five random seeds (Left) and real tensors from Llama3.2-3B’s block.11.feed_forward.w2 (Right). Improvement is defined as

(E_{\mathrm{base}}-E_{\mathrm{IHT}})/E_{\mathrm{base}}\times 100

; see Appendix D for the full setup and error definitions.

Figure 4: Outlier patterns across 300 training steps for Llama3.2-3B. Patterns remain stable throughout training, enabling one-time calibration. Other layers in Appendix C .

Figure 5: AdaHOP pipeline. For each linear layer’s three matrix multiplications, AdaHOP selectively applies IHT, OE (Left or Right), or BF16 based on the detected outlier pattern pair $P_{n}$ ( $n=1,3,5$ ).

Model	Tokens	Method	PIQA	HellaSwag	ARC-E	Lambada	Avg.
Llama3.2-1B	40B	BF16	72.69	52.33	47.85	38.17	52.76
		MXFP4	64.69	37.94	37.29	11.93	37.96
		MXFP4+Had.	71.85	51.37	45.91	37.68	51.70
		Tseng et al.	72.14	50.95	46.17	36.81	51.52
		HALO	72.09	51.89	46.72	39.03	52.43
		AdaHOP-Lv1	73.04	52.24	46.91	38.21	52.60

Table 2: Zero-shot accuracy (%) after training on C4. Best MXFP4-based result is in bold ; second best is underlined .

Method	Mem.	Speed	Accel.	Quality
BF16	76.00	10930	1.00	✓
MXFP4	20.19	17640	1.61	✗
MXFP4+Had.	20.60	16305	1.49	✗
Tseng et al.	20.60	14881	1.36	▲
HALO	20.60	14251	1.30	▲
AdaHOP-Lv1	20.94	15804	1.45	✓

Table 3: (Left) End-to-end efficiency on Llama3.1-8B training with seq_len=4096 and BS=8 . Memory (GB), Speed (Token/sec), and acceleration (

\times

) are measured for the full training pipeline. Quality: ✓ within 0.3pt of BF16, ▲ within 1pt, ✗ otherwise, based on the average downstream score across all models. (Right) kernel latency (ms) breakdown for two representative Llama3.1-8B GEMMs.

Figure 6: Throughput per batch size on Llama3.1-8B with seq_len=4096 . Highlighted number is the speedup of AdaHOP-Lv2. As the batch size increases, acceleration is maximized.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Per-column mean magnitude, variance, and quantization error of the activation tensor from Llama3.2-3B’s block.11.feed_forward.w2 . Red bars indicate the top-32 columns for each metric; light blue bars indicate the remaining columns.

Figure 8: Outlier patterns of weight (

W

), activation (

X

), and gradient (

G_{Y}

) tensors across 300 training steps for all 12 representative blocks of Llama3.2-3B. Each row represents a tensor from a specific layer, and the color indicates the detected outlier pattern at each step. This extended view reveals depth-dependent transitions in gradient outlier patterns: Row-wise patterns appear in early blocks, fade to None in the middle layers, and re-emerge selectively in the K-projection gradient toward the final blocks.

Figure 9: Hardware-aware implementation pipeline of AdaHOP. The pipeline consists of four stages: FOID, IHT and Quantization, Fused MXFP4+BF16 GEMM, and Fused Scatter-Add, all implemented as optimized Triton kernels targeting the AMD CDNA4 architecture. Note that the red block in Fast Outlier Index Detection represents the first 64 elements along each column.

Figure 10: MSE and latency as a function of the OE extraction rank for (Left) real tensors from Llama3.1-8B and (Right) synthetic tensors. Rank 64 achieves a favorable accuracy–efficiency trade-off in both settings: MSE is substantially reduced while latency remains close to the minimum.

Walsh-Hadamard Transform

See all 11 papers

HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization
May 28, 2026Artur Zagitov, Gleb Molodtsov, Aleksandr BeznosikovLLM Quantization Rotation-Based Quantization
1MIRAI · 2BRAIn Lab · 3Innopolis University
M+Adam: Low-Precision Training via Additive-Multiplicative Optimization
Jul 12, 2026Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1Quantization-Aware Training Low-Precision Training
California Institute of Technology · University of Copenhagen · Aarhus University
The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Mar 11, 2026Hengjie Cao, Zhendong Huang, Mengyi Chen +15LLM Quantization 4-Bit Quantization
Fudan University · Shanghai Innovation Institute · University of Bath +5

May 28, 2026J/K move · Enter open · S save

HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization

Artur Zagitov, Gleb Molodtsov, Aleksandr Beznosikov

1MIRAI · 2BRAIn Lab · 3Innopolis University

LLM Quantization Rotation-Based Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) is essential for deploying LLMs under memory and bandwidth constraints. However, extreme low-bit quantization remains highly sensitive to activation outliers and anisotropic weight curvature. Existing incoherence-based PTQ methods mitigate this issue with fixed randomized Hadamard transforms (RHTs), which improve quantization robustness but cannot adapt the rotated basis to the layer, calibration distribution, or quantizer. We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided orthogonal processor that replaces fixed Hadamard mixing while preserving exact full-precision equivalence. HARP represents each rotation as a product of sparse butterfly-like block-orthogonal stages, supports non-power-of-two dimensions through Mixed-Radix schedules, and initializes to the RHT processor up to a fixed permutation. Fitted only on calibration data, HARP adapts the quantization basis to each layer and backend. Across 2--4-bit settings on Llama models from 1B to 70B, HARP consistently improves perplexity and yields its clearest zero-shot gains at 2 bits; a 2-bit Qwen3-8B experiment shows the same transfer beyond the Llama family. HARP also preserves deployment efficiency: on Llama 2 7B at 2 bits, it reaches 128 tok/s, retaining 90% of RHT throughput (142 tok/s) and running approximately

2.1\times

faster than FP16 (61 tok/s).

Abstract

Walsh-Hadamard Transform

Explore similar work

HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization

M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training