cs.LGOct 31, 2026

Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention

Authors: Haozhan Tang, Hao Kang, Meng Lu, Yecheng Wu, Zerui Wang, Seokho Han, Haocheng Xi, Shuo Yang, 

Organizations: Carnegie Mellon University · Virginia Tech · Massachusetts Institute of Technology · NVIDIA · 

Abstract

Long-horizon reasoning and agentic workloads strain KV-cache capacity and decode attention throughput. We present Jet-QSparse, a training-free method that combines compact NVFP4 cache storage with native FP4 query-key computation on Blackwell tensor cores. A dense scan scores the full key history and selects 128-token blocks for sparse FP16 value aggregation with FP32 accumulation, preserving higher precision in the output-forming step while enabling contiguous tile loads. Calibrated key centering and at most two BF16 pages per sequence and layer address quantization error; FP8 prefill uses a temporary arena while the persistent cache is packed directly from BF16 activations. Across four hybrid LLMs, a budget of 128 blocks per KV head shows no detected accuracy loss on the evaluated mathematical reasoning, coding, and agentic benchmarks, although some smaller budgets incur losses. With 64 blocks, Qwen3.8-27B achieves a 1.94×1.94\times end-to-end decode speedup over dense BF16 at 128K context and batch size 16 on one GB300, and the measured Qwen KV pools provide 2.582.58-2.76×2.76\times BF16's token capacity.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    Jun 25, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +44-Bit QuantizationLLM Inference Acceleration

  2. UltraQuant: 4-bit KV Caching for Context-Heavy Agents

    Jun 18, 2026Inesh Chakrabarti, David Limpus, Aditi Ghai Rana +4LLM Inference EfficiencyGPU Kernel Optimization