Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition f(g(x)), we provide an adaptive strategy that selects a small subset of components of g(x) to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.
Figures & tables
Figure 1: Performance of mixed-precision GPT-2 XL inference on the OpenWebText dataset with the proposed LAMP evaluation of KQ inner products. Only 0.9% of them are recomputed in FP32. The Kullback–Leibler divergence is measured against a reference model with uniform FP32 accumulation.
Figure 2: Performance of mixed-precision GPT-2 XL inference on the OpenWebText dataset with LAMP evaluation of KQ inner products.
Figure 3: Comparison of Pareto boundaries of “strict” LAMP ( 8 ) and relaxed LAMP ( 9 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on different datasets.
Figure 5: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL and GPT-2 small with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Figure 6: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset with direct and permuted tokens.
Figure 7: Comparison of Pareto boundaries of LAMP ( 8 ) and its analogue with randomly selected recomputations applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Dataset
Method
Specification
Perplexity
Sparsity
GSM8k Math
Full precision
N/A
13.088
100%
Low precision
N/A
13.161
0%
LAMP
Relaxed ( τ=0.03 )
13.091
5.8%
Relaxed LN ( τ=0.03 )
13.094
4.5%
Relaxed ( τ=0.09 )
13.093
2.8%
Relaxed LN ( τ=0.09 )
13.096
2.0%
Appendix
Table 1: Comparison of perplexities of full-precision inference, low-precision inference, relaxed LAMP ( 9 ), and its length-normalized modification applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on several datasets.
Transformer-based large language models (LLMs) have demonstrated remarkable performance across a wide range of real-world tasks, but their inference cost remains prohibitively high due to the quadratic complexity of attention and the memory bandwidth limitations of high-precision operations. In this work, we present a low-bit mixed-precision attention kernel using the microscaling floating-point (MXFP) data format, utilizing the computing capability on next-generation GPU architectures. Our Diagonal-Tiled Mixed-Precision Attention (DMA) incorporates two kinds of low-bit computation at the tiling-level, and is a delicate fused kernel implemented using Triton, exploiting hardware-level parallelism and memory efficiency to enable fast and efficient inference without compromising model performance. Extensive empirical evaluations on NVIDIA B200 GPUs show that our kernel maintains generation quality with negligible degradation, and meanwhile achieves significant speedup by kernel fusion. We release our code at https://github.com/yifu-ding/MP-Sparse-Attn.
Yifu Ding, Xinhao Zhang, Jinyang Guo
State Key Laboratory of Complex & Critical Software Environment (SKLCCSE), Beihang University · School of Computer Science and Engineering, Beihang University · Beijing Jiaotong University +1
Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.
Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies
Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost. This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth. Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory. We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation. Matched-budget experiments strongly support this prediction, showing that placing quantized computation earlier in recurrent depth consistently yields the best fidelity. Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement. Across context lengths up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16x end-to-end first-prefill speedup on an NVIDIA H100, corresponding to a 13.9% latency reduction.
Ke Wan, Chen Chen
Department of Computer Science, University of Virginia