cs.AINov 6, 2026

FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding

Authors: Donglin Zhuang, Pragaash Ponnusamy, Xiaoxia Wu, Shuaiwen Leon Song, Zhongzhu Zhou

Organizations: Together AI · University of Sydney

Abstract

Large-language-model decode attention streams the key-value (KV) cache for every generated token. Packing that cache into 2-bit or 4-bit quantized integers reduces memory traffic but does not automatically translate into latency saving. Existing 2-bit fused kernels reach under 31%/15% of peak HBM throughput on H100/B200. Converting each value and applying its scale and offset before feeding Tensor Cores is a major bottleneck because instruction-issue throughput is scarce on modern GPUs. Instead of applying scale and offset to each KV value, FOSCAR uses affine folding to apply them to the other operand or the output of decode-attention matrix products, depending on the quantization scheme (token-wise or channel-wise). This reduces scale-and-offset operations by up to two orders of magnitude while preserving the dequantized attention output in exact arithmetic. FOSCAR also uses bit manipulation to feed KV operands to Tensor Cores efficiently, requiring only 1 and 0.19 ALU instructions per INT2 element on H100 and B200, respectively. Finally, FOSCAR builds warp-specialized GPU pipelines for Hopper (H100) and Blackwell (B200) using wgmma and tcgen05. On H100, FOSCAR INT2 and INT4 kernels achieve geo-mean speedups of 3.36×3.36\times and 2.83×2.83\times over FlashInfer BF16, and 2.03×2.03\times and 1.71×1.71\times over TensorRT-LLM FP8. On B200, the corresponding INT2 and INT4 speedups are 2.67×2.67\times and 2.30×2.30\times over FlashInfer BF16, 1.48×1.48\times and 1.28×1.28\times over TensorRT-LLM FP8, and 1.16×1.16\times and 1.00×1.00\times over TensorRT-LLM NVFP4. We integrate FOSCAR into SGLang to benchmark serving efficiency. On long-context serving workloads, FOSCAR increases peak aggregate throughput by up to 170% and 341% over 2-bit and 16-bit baselines, respectively. We open-source FOSCAR as a kernel library at https://github.com/FutureMLS-Lab/FOSCAR.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

    May 18, 2026Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4Rotation-Based QuantizationLLM Inference Acceleration

  2. BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

    Nov 6, 2026Dayou Du, Woon Yee Ng, Guangming Sheng +3LLM Inference AccelerationKV-Cache Quantization