Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition f(g(x)), we provide an adaptive strategy that selects a small subset of components of g(x) to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.
Figures & tables
Figure 1: Performance of mixed-precision GPT-2 XL inference on the OpenWebText dataset with the proposed LAMP evaluation of KQ inner products. Only 0.9% of them are recomputed in FP32. The Kullback–Leibler divergence is measured against a reference model with uniform FP32 accumulation.
Figure 2: Performance of mixed-precision GPT-2 XL inference on the OpenWebText dataset with LAMP evaluation of KQ inner products.
Figure 3: Comparison of Pareto boundaries of “strict” LAMP ( 8 ) and relaxed LAMP ( 9 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on different datasets.
Figure 5: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL and GPT-2 small with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Figure 6: Comparison of Pareto boundaries of LAMP ( 8 ) applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset with direct and permuted tokens.
Figure 7: Comparison of Pareto boundaries of LAMP ( 8 ) and its analogue with randomly selected recomputations applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on the OpenWebText dataset.
Dataset
Method
Specification
Perplexity
Sparsity
GSM8k Math
Full precision
N/A
13.088
100%
Low precision
N/A
13.161
0%
LAMP
Relaxed ( τ=0.03 )
13.091
5.8%
Relaxed LN ( τ=0.03 )
13.094
4.5%
Relaxed ( τ=0.09 )
13.093
2.8%
Relaxed LN ( τ=0.09 )
13.096
2.0%
Appendix
Table 1: Comparison of perplexities of full-precision inference, low-precision inference, relaxed LAMP ( 9 ), and its length-normalized modification applied to GPT-2 XL with μ=4 mantissa bits for KQ accumulation and validated on several datasets.
State Key Laboratory of Complex & Critical Software Environment (SKLCCSE), Beihang University · School of Computer Science and Engineering, Beihang University · Beijing Jiaotong University +1