Organizations: LLaVi Lab, Computer Science & Engineering, University of North Texas. · Computer Science, Saint Louis University. · Data Science, University of North Texas.
Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interactions differ in their precision requirements: some tolerate low-precision scores while remaining sensitive to removal. We therefore introduce TriAttention, which assigns query-key (QK) tiles to skipping, 4-bit, or 8-bit computation, using 4-bit to retain quantization-tolerant interactions at lower cost. To represent compactly, we encode each head's computation regions with two diagonal-distance boundaries, keeping policy storage independent of context length. Offline calibration then evaluates candidate boundary pairs using normalized projected-output errors and complete-tile execution costs. Using these measurements, a model-wide allocator selects each head's boundaries to minimize calibration error under a shared computation budget. During prefill, the selected boundaries generate precision-grouped traversals for a fused kernel that skips omitted tiles and combines contributions from both QK precisions into one normalized attention output. The quantized keys and values prepared during prefill are then reused in a compressed key-value (KV) cache for decoding, reducing repeated reads of prompt data. Experiments show that TriAttention achieves end-to-end generation speedups of up to 7.88x over the Torch attention and 2.65x over the BF16 baseline. Meanwhile, TriAttention maintains quality comparable to the original models on general-purpose and long-context tasks. These results show how a compact retention-and-precision policy translates local sensitivities into attention computation and repeated KV-cache access. The implementation is https://github.com/HanzhiZhang-Ulrica/TriAttention.
Figures & tables
Figure 1: Attention computation choices and inference performance. Top left: attention maps and token-interaction graphs compare dense, sparse, quantized, and sparse-quantized designs with TriAttention , which jointly assigns skipping, 4-bit, and 8-bit. Bottom left: LooGLE single-hop QA F1 at 64K. Right: throughput across input lengths. Both evaluations use Llama-3.2-3B-Instruct.
Figure 2: Offline calibration and compact distance encoding. Left: candidate boundary pairs are evaluated by normalized projected-output error. Center: errors and complete-tile execution costs guide allocation under a shared computation budget. Full-model prediction checks select the final boundary table. Right: two diagonal-distance boundaries per layer and query head encode 8-bit, 4-bit, and skipped QK regions, with the first key block and diagonal tiles protected.
Figure 3: Per-layer QK sensitivity in Llama-3.2-3B-Instruct. Rows are query heads; columns are diagonal-distance bands ordered near to far. Colors show layer-normalized projected-output error from 4-bit QK or skipping, with nearer bands kept in 8-bit.
Figure 5: Fused prefill with compact, precision-grouped traversal. The illustrated query block visits protected tiles 0 and 6, 8-bit QK tiles 4 and 5, and 4-bit QK tile 3. Protected tiles use 16-bit PV, while other retained tiles use 8-bit PV. Shared-memory staging and query reuse feed one register-resident attention state, followed by normalization, value restoration, and 16-bit output.
Figure 6: Mixed-precision decode over the compressed prompt KV cache. Alternating shared-memory buffers B0/B1 stage retained KV tiles for 8-bit (blue) or 4-bit (green) computation. The kernel reuses query fragments and maintains attention state in registers across tiles. Each key segment writes partial states for subsequent merging with other segments and the 16-bit generated-token tail.
Figure 7: Generation-throughput ratio on A100-SXM4-40GB. Llama-3.2-3B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Table 3: LV-Eval QA results for Llama-3.2-3B-Instruct. Dataset identifiers follow Section 5.4 . Scores use official task-specific metrics on a 0–100 scale; higher is better. Length labels follow the official dataset levels. BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours, Others, and Ablations; displayed ties share the mark.
Model
Context
TriAttention (s)
Without 4-bit (s)
Without Skip (s)
Llama
32K
0.654
0.660
0.682
64K
2.242
2.259
2.647
128K
7.347
7.568
10.324
Qwen2
32K
0.781
0.810
0.845
64K
2.752
2.924
3.232
128K
10.134
11.151
13.010
Table 4: Effect of three-level computation on A100 prefill attention latency. Latency includes Q/K preparation, V packing, and attention across all layers. Ablations are independently reallocated under the same quality-selection thresholds. Lower is better.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Context
8-bit PV (ms)
16-bit PV (ms)
Llama
32K
72.9
79.2
64K
251.7
275.3
128K
865.0
992.7
Qwen2
32K
77.1
83.6
64K
269.4
295.1
128K
985.2
1,089.4
Appendix
Table 5: Effect of value-aggregation precision on A100 prefill latency. The comparison covers the first, middle, and last attention layers with identical retained tiles and 8-bit QK computation. Non-protected tiles use either 8-bit or 16-bit PV. Lower is better.
Model
Context
KV-Group-Major (s)
Heaviest-First (s)
Llama
32K
0.654
0.681
64K
2.242
2.359
128K
7.347
7.803
Qwen2
32K
0.781
0.782
64K
2.752
2.790
128K
10.134
10.339
Appendix
Table 7: Effect of KV-group-aware CTA scheduling on A100 prefill attention latency. Both schedules execute the same selected tiles and arithmetic operations, producing bitwise-identical outputs. KV-group-major scheduling places query heads sharing a KV head together to improve K/V data reuse. Lower is better.
Model
Context
BF16 (GiB)
Full-Retention Compressed (GiB)
TriAttention (GiB)
Llama
32K
3.50
1.33
1.21
64K
7.00
2.64
2.10
128K
14.00
5.27
3.74
Qwen2
32K
1.75
0.66
0.66
64K
3.50
1.32
1.28
128K
7.00
2.64
2.51
Appendix
Table 8: Prompt KV-cache storage after prefill. The full-retention compressed cache stores all prompt tiles with 8-bit keys and mixed-bit values. TriAttention additionally applies its selected retention and key-precision policy. Sizes exclude model weights and temporary inference workspace. Lower is better.
Figure 8: TPOT speedup on A100-SXM4-40GB. Llama-3.2-3B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 9: Generation-throughput ratio on A100-SXM4-40GB. Qwen2-7B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 10: TPOT speedup on A100-SXM4-40GB. Qwen2-7B-Instruct. Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 10: Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.
Figure 11: Generation-throughput ratio on A100-SXM4-40GB. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 12: TPOT speedup on A100-SXM4-40GB. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 13: Generation-throughput ratio on H100 NVL. Llama-3.2-3B-Instruct (prompt group 1 of 2). Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 13: Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.
Figure 14: Generation-throughput ratio on H100 NVL. Llama-3.2-3B-Instruct (prompt group 2 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 16,384, 32,768 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. The 128-token column repeats the first group as a shared reference. Panels share colors and order.
Figure 15: TPOT speedup on H100 NVL. Llama-3.2-3B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 16: TPOT speedup on H100 NVL. Llama-3.2-3B-Instruct (prompt group 2 of 2). Rows use batch sizes 1 and 2; columns use 128, 16,384, 32,768 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. The 128-token column repeats the first group as a shared reference. Method colors and order are fixed across panels.
Figure 16: Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.
Figure 17: Generation-throughput ratio on H100 NVL. Qwen2-7B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 18: TPOT speedup on H100 NVL. Qwen2-7B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 19: Generation-throughput ratio on RTX 4090. Llama-3.2-3B-Instruct. Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 19: Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.
Figure 21: Generation-throughput ratio on H100 NVL. Qwen2-7B-Instruct (prompt group 2 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 16,384, 32,768, 65,536 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. The 128-token column repeats the first group as a shared reference. Panels share colors and order.
Figure 23: Generation-throughput ratio on H100 NVL. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1× ; higher is better. Method colors and order are fixed across panels.
Figure 30
Table 10: General-task results for Qwen2-7B-Instruct. Accuracy is reported in percent (higher is better); WikiText-2 uses word perplexity (lower is better). BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours and Others; displayed ties share the mark.
Table 12: General-task results for Ministral-3-8B-Instruct-2512-BF16. Accuracy is reported in percent (higher is better); WikiText-2 uses word perplexity (lower is better). BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best of Ours and Others, including displayed ties.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Quantization has been widely adopted in LLM training and inference to reduce cost and improve efficiency. However, low-bit quantization of the attention module often introduces large errors at very low bit-widths, causing performance degradation. Existing methods mainly rely on smoothing techniques to handle outliers, while we propose a hybrid quantization design to better balance accuracy and efficiency. Specifically, we propose HyQuant, an efficient hybrid quantization framework for LLM attention. HyQuant quantizes most attention states into low-bit formats while retaining a small set of vertical-line tokens and local-window states in high precision. These accuracy-critical regions are selected using lightweight vertical-line-aware attention-pattern signals, reducing quantization error with limited overhead. In the Prefill stage, HyQuant uses a hybrid-precision quantized attention operator that preserves vertical-line tokens and a local sliding window in full precision while quantizing the remaining context. In the Decode stage, HyQuant applies the same principle to KV-cache compression and fuses KV dequantization with attention computation to improve memory and hardware efficiency. Across diverse tasks, models, and datasets, HyQuant maintains nearly lossless accuracy with an extremely simple design, demonstrating the efficiency and practical feasibility of hybrid quantization for LLM attention. Code is available at: https://github.com/jerrysfls/HyQuant .
Jiatong Ding, Bingxin Xing, Yu Zhang +9
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
Efficient attention algorithms are critical to mitigate the quadratic cost of attention in long-context workloads. Prior work utilises block-scaled quantisation techniques on Blackwell GPUs to move attention computation to 4-bit precision to accelerate inference. However, these techniques result in significant quality degradation in long-context settings. We show that the output impact of quantisation error is highly non-uniform and increases with the importance of each query-key interaction, concentrating functionally relevant error in a small number of attention blocks that contain the most important tokens. We propose ThriftAttention, a low-bit attention variant that delivers near-FP16 long-context quality at FP4 inference efficiency. This approach proceeds in two stages. First, a heuristic rapidly selects a small number of important query-key block pairs for FP16 precision. Second, the selected blocks are computed in FP16 and the remaining blocks in FP4, with both paths merged via online softmax into a single output. We demonstrate across long-context benchmarks and model families that by computing only 5% of query-key blocks in FP16, ThriftAttention recovers on average 89.1% of the FP4-to-FP16 performance gap. We show ThriftAttention's advantage grows with sequence length, mitigating the systematic FP4 quality degradation observed at longer contexts. The code is available at https://github.com/joesharratt1229/ThriftAttention.
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.
Wenhao Li, Jinhao Dong, Hailin Zhang +3
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China