LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

Stanislav Budzinskiy; Marian Gloser; Tolunay Yilmaz; Ying Hong Tham; Yuanyi Lin; Wenyi Fang; Fan Wu; Philipp Petersen

cs.LGSep 30, 2026

LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

Authors: Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz, Ying Hong Tham, Yuanyi Lin, Wenyi Fang, Fan Wu, Philipp Petersen

Organizations: Faculty of Mathematics University of Vienna, Austria · Huawei Heisenberg Research Center, Munich, Germany · Huawei Technologies Co. Ltd

Abstract

While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.

Figures & tables

	Perplexity $\bm{(\downarrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	21.470	17.150	16.540	46.30%	20.52%	33.18%
gemma-3-12b-pt	36.740	21.830	18.680	50.94%	20.30%	28.76%
Qwen3-32B	28.180	23.740	23.800	65.09%	15.43%	19.48%
Qwen3-8B	44.030	33.780	33.340	66.19%	14.58%	19.23%
Qwen3-30B-A3B	43.900	28.270	27.510	69.75%	13.71%	16.54%

Table 1: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on C4.

	Accuracy $\bm{(\uparrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	0.6261	0.7851	0.7993	48.49%	21.55%	29.96%
gemma-3-12b-pt	0.3618	0.5899	0.7654	53.39%	22.19%	24.42%
Qwen3-32B	0.7555	0.8355	0.8344	54.52%	19.67%	25.81%
Qwen3-8B	0.5625	0.7412	0.7840	54.05%	19.11%	26.84%
Qwen3-30B-A3B	0.5998	0.7884	0.8169	56.97%	18.99%	24.04%

Table 2: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on 5-shot MMLU.

Figure 1: LampAttention with

\delta=2^{-8}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Figure 2: LampAttention with

\delta=2^{-8}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Perplexity $\bm{(\downarrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	8.6290	6.2100	5.9000	47.21%	19.86%	32.93%
gemma-3-12b-pt	15.940	8.6370	7.3790	52.86%	19.10%	28.04%
Qwen3-32B	12.170	9.4290	9.3610	76.50%	11.55%	11.95%
Qwen3-8B	17.480	12.510	12.300	76.44%	11.01%	12.55%
Qwen3-30B-A3B	18.780	11.260	10.890	81.82%	9.02%	9.16%

Appendix

Table 3: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on Wikitext.

	Accuracy $\bm{(\uparrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	0.5977	0.6484	0.6562	48.47%	30.41%	21.12%
gemma-3-12b-pt	0.4570	0.6191	0.6387	48.71%	32.21%	19.08%
Qwen3-32B	0.6191	0.6016	0.6094	48.54%	31.22%	20.24%
Qwen3-8B	0.4766	0.5352	0.5605	48.24%	28.71%	23.05%
Qwen3-30B-A3B	0.4512	0.5176	0.5762	48.28%	28.94%	22.78%

Appendix

Table 4: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on 0-shot ARC-Challenge.

Figure 3: LampAttention with

\delta=2^{-8}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Figure 4: LampAttention with

\delta\in\{2^{-8},2^{-4}\}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Figure 5: LampAttention with

\delta\in\{2^{-8},2^{-4}\}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Figure 6: LampAttention with

\delta=2^{-8}

or disabled first stage, and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Figure 7: LampAttention with

\delta=2^{-8}

or disabled first stage, and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

	Accuracy $\bm{(\uparrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	0.6504	0.6953	0.7070	56.64%	19.37%	23.99%
gemma-3-12b-pt	0.4609	0.6289	0.6777	62.56%	19.31%	18.13%
Qwen3-32B	0.6758	0.7305	0.7324	55.23%	19.55%	25.22%
Qwen3-8B	0.5488	0.6660	0.6660	54.10%	18.53%	27.37%
Qwen3-30B-A3B	0.5879	0.7031	0.6953	57.72%	19.01%	23.27%

Appendix

Table 5: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on 25-shot ARC-Challenge.

	Accuracy $\bm{(\uparrow)}$			16-bit sub-blocks per tile
Model name	8-bit	8/16-bit LAMP	32-bit	0	1	2+
gemma-3-27b-pt	0.5977	0.6484	0.6562	48.47%	30.41%	21.12%
gemma-3-12b-pt	0.4570	0.6191	0.6387	48.71%	32.21%	19.08%
Qwen3-32B	0.6191	0.6016	0.6094	48.54%	31.22%	20.24%
Qwen3-8B	0.4766	0.5352	0.5605	48.24%	28.71%	23.05%
Qwen3-30B-A3B	0.4512	0.5176	0.5762	48.28%	28.94%	22.78%

Appendix

Table 6: LampAttention with

\delta=2^{-8}

and

\tau=2^{-1}

on 0-shot ARC-Challenge.

Figure 8: LampAttention with

\delta=2^{-8}

and

\tau\in\{2^{-t}~:~t=0,\ldots,5\}

Explore similar work

Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3Block Sparse Flash Attention Efficient Long-Context Inference
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeBlock Sparse Flash Attention Quantized
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Large Language Model Quantization Qantis
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1

Dec 7, 2025, cs.LGJ/K move · Enter open · S save

Block Sparse Flash Attention

Daniel Ohayon, Itay Lamprecht, Itay Hubara, Israel Cohen+2

Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel

Block Sparse Flash Attention Efficient Long-Context Inference LLM Inference Optimization Bottlenecks

Open paper AskPDF arXiv

Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottleneck. We present Block Sparse Flash Attention (BSFA), a drop-in replacement that accelerates long-context inference while preserving model quality. Unlike methods that predict importance before computing scores, BSFA computes exact query-key similarities to select the top-k most important value blocks for each query. By comparing per-block maximum scores against calibrated thresholds, we skip approximately 50% of the computation and memory transfers for pruned blocks. Our training-free approach requires only a one-time threshold calibration on a small dataset to learn the per-layer and per-head attention score distributions. We provide a CUDA kernel implementation that can be used as a drop-in replacement for FlashAttention. On Llama-3.1-8B, BSFA achieves up to 1.13x end-to-end speedup on LongBench with only a 1.1% accuracy drop, and up to 1.24x on Needle-in-a-Haystack retrieval at a 1% accuracy drop. The attention kernel itself accelerates by up to 1.38x. We compare BSFA against five recent sparse attention baselines (SpargeAttention, MInference, FlexPrefill, XAttention, and BLASST), and verify the method on Qwen2.5-7B and on A6000 and H100 GPUs. The implementation is available at https://github.com/Danielohayon/Block-Sparse-Flash-Attention.

Abstract

Explore similar work

Block Sparse Flash Attention

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

HyQuant: Hybrid-Precision Quantization for LLM Attention