Inference with transformer models on CPUs is increasingly important, especially for Small Language Models (SLMs), where vector architectures are emerging as a promising execution substrate. The attention module is a major bottleneck due to high memory bandwidth requirements; FlashAttention mitigates this by fusing operations to improve data locality and reduce intermediate memory traffic. In this paper, we present FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality. We integrate FlashAttention-V into ggml within llama.cpp and evaluate it on TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M using gem5 and a Banana Pi BPI-F3. On the Banana Pi BPI-F3, we confirm that loop reordering and loop unrolling across attention heads are effective optimization principles, scaling performance gains with larger models and most pronounced with short contexts and during decoding. Simulation-based analysis shows that FlashAttention-V achieves 22x-42x speedup over scalar FlashAttention at 512-bit VL in prefill, with an additional 2x-2.5x gain scaling to 64 lanes and 4096-bit VL. During decode, FlashAttention-V achieves 8x-11x speedup using 512-bit vector lengths over scalar FlashAttention, with performance showing diminishing sensitivity to vector width and lane count due to single-token, memory-bound execution. We further identify structural bottlenecks in Q8_0 quantized linear layers that limit arithmetic amortization under long-vector execution, consistent across RVV and Arm SVE, indicating that current quantization formats pose a fundamental challenge to long-vector scalability.
Figures & tables
Figure 1. Inter-head packing into a single vector register to improve vector utilization across attention heads with FP16
Component
Specification
Core frequency
2GHz
Vector length
512-bits to 8192-bits
Memory type
DDR3-1600
Memory bandwidth
12.8 GB/s per core
Cache hierarchy
L1: 64 kB; L2: 1 MB
Table 1. Simulated system configuration
Op Class
MinorCPU
O3_ARM
VL-Scaled
Fixed (cycles)
base (cycles)
SimdFloatAdd
6
5
Yes
SimdFloatAlu
6
5
Yes
SimdFloatMult
6
3
Yes
SimdFloatMultAcc
6
5
Yes
SimdFloatReduceAdd
6
4
Yes
Table 2. Comparison of MinorCPU fixed latencies, and ARM OoO base latencies (O3_ARM_v7a.py), with VL-scaling indicated for each operation class.
Model
Attention
KV
Hidden
Head
Size
Variants
Heads
Heads
Size
Dimension
(D)
TinyLlama
32
4
2048
64
1.1B
Llama 3.2
32
8
2048
64
1B
Qwen2.5
16
2
2048
128
3B
Pythia
16
16
1024
64
410M
Table 3. Dimensions of Decoder-only LLMs Used in Our Evaluation
Figure 2. Prefill Performance Comparison of FlashAttention-V and ggml-vec-fp16
Version
Prefill stage
Decode stage
Non-vectorized
21.38
0.000529
Vectorized by ggml with FP32
3.67
0.000300
FlashAttention-V with FP32
0.89
0.000077
FlashAttention-V with FP16
0.78
0.000070
Table 4. Execution time for 1 FlashAttention using gem5@RVV with 512-bits VL for TinyLlama
Figure 3. Prefill stage: Speedup across different vector lengths using gem5@RVV for TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M with 512 tokens, compared to ggml-scalar
Figure 4. Prefill speedup across vector lengths (gem5@RVV, 512 tokens) for TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M, with VL-proportional operation latencies per vector lane, normalized to ggml-scalar .
Figure 5. Decode stage: Speedup across vector lengths (VLs) on gem5@RVV for TinyLlama, Llama 3.2, Qwen 2.5, and Pythia-410M, relative to ggml-scalar
Figure 6. Decode stage: Speedup across different vector lengths (X-axis) using gem5@RVV for TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M, considering operation latencies per vector lane (baseline: ggml-scalar )
Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottleneck. We present Block Sparse Flash Attention (BSFA), a drop-in replacement that accelerates long-context inference while preserving model quality. Unlike methods that predict importance before computing scores, BSFA computes exact query-key similarities to select the top-k most important value blocks for each query. By comparing per-block maximum scores against calibrated thresholds, we skip approximately 50% of the computation and memory transfers for pruned blocks. Our training-free approach requires only a one-time threshold calibration on a small dataset to learn the per-layer and per-head attention score distributions. We provide a CUDA kernel implementation that can be used as a drop-in replacement for FlashAttention. On Llama-3.1-8B, BSFA achieves up to 1.13x end-to-end speedup on LongBench with only a 1.1% accuracy drop, and up to 1.24x on Needle-in-a-Haystack retrieval at a 1% accuracy drop. The attention kernel itself accelerates by up to 1.38x. We compare BSFA against five recent sparse attention baselines (SpargeAttention, MInference, FlexPrefill, XAttention, and BLASST), and verify the method on Qwen2.5-7B and on A6000 and H100 GPUs. The implementation is available at https://github.com/Danielohayon/Block-Sparse-Flash-Attention.
Daniel Ohayon, Itay Lamprecht, Itay Hubara +3
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison. To address these challenges, we propose QFlash, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel. On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73× speedup over I-ViT and up to 8.69× speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization. Our code is publicly available at https://github.com/EfficientCompLab/qflash.
Sehyeon Oh, Yongin Kwon, Jemin Lee
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. While FlashAttention-3 optimized attention for Hopper GPUs through asynchronous execution and warp specialization, it primarily targets the H100 architecture. The AI industry has rapidly transitioned to deploying Blackwell-based systems such as the B200 and GB200, which exhibit fundamentally different performance characteristics due to asymmetric hardware scaling: tensor core throughput doubles while other functional units (shared memory bandwidth, exponential units) scale more slowly or remain unchanged. We develop several techniques to address these shifting bottlenecks on Blackwell GPUs: (1) redesigned pipelines that exploit fully asynchronous MMA operations and larger tile sizes, (2) software-emulated exponential and conditional softmax rescaling that reduces non-matmul operations, and (3) leveraging tensor memory and the 2-CTA MMA mode to reduce shared memory traffic and atomic adds in the backward pass. We demonstrate that our method, FlashAttention-4, achieves up to 1.3× speedup over cuDNN 9.13 and 2.7× over Triton on B200 GPUs with BF16, reaching up to 1613 TFLOPs/s (71% utilization). Beyond algorithmic innovations, we implement FlashAttention-4 entirely in CuTe-DSL embedded in Python, achieving 20-30× faster compile times compared to traditional C++ template-based approaches while maintaining full expressivity.