Kwai Summary Attention V2 Technical Report

OneRec Team; Chenglong Chu; Guohong Mu; Guowang Zhang; Hao Peng; Hui Wang; Jian Liang; Jiangxia Cao

cs.LGOct 13, 2026

Kwai Summary Attention V2 Technical Report

Authors: OneRec Team, Chenglong Chu, Guohong Mu, Guowang Zhang, Hao Peng, Hui Wang, Jian Liang, Jiangxia Cao,

Abstract

Summary-token architectures such as Kwai Summary Attention (KSA) compress distant context into learnable Summary token per chunk, and hybrid variants interleave these layers with full attention to retain exact access to the original text. In such hybrid KSA-Full architecture, the interleaving inserted summary tokens are served as a streaming flow role across different layers. Although effective, however, it also introduce some bottlenecks: (1) the different layers summary compression chunk must be same, therefore hard to reach a flexible KV Cache compression, e.g., different layer set different summary compression chunk window. (2) in the full-attention layers, the summary token introduced additional computation cost, but such layer ignore them and still pay quadratic prefilling, therefore unfriendly to the computation side. Motivated by them, we developed the KSA V2, which not only achieves a more flexible KV cache memory compression but also reduces the computation cost in Full attention layer. Specifically, we found that the summary tokens can serve two roles: the KV memory and sparse indexer.

Figures & tables

Figure 1: Left: long-context and general capabilities of 1.9B C-KSA+S-KSA and the reference models reported in KSA. Right: per token inference FLOPs and cache size (

1\mathrm{GiB}=2^{30}

bytes).

Figure 2: Architecture overview: from Hybrid KSA to KSA V2.

Figure 3: Compressed KSA with

k_{s}=4

and

k_{m}=2

. Complete remote Summary groups are compressed into Meta KV, while boundary Summary KV and local text KV remain directly accessible.

Figure 4: Sparse KSA with

k_{s}=4

and

b=2

Panel of Figure 5 — Figure 5: C-KSA cache layout and decoding updates with $k_{s}=2$ , $W=4$ , and $k_{m}=4$ . Gray ( $T$ ), orange ( $S$ ), and purple ( $M$ ) denote text, Summary, and Meta KV. Blue marks indicate reads or writes.

Mechanism	Effective Context	KV Cache Size ( $N\rightarrow\infty$ )	Compression Rate
MHA	$N$ (exactly)	$2\cdot N\cdot H\cdot d_{h}$	—
GQA	$N$ (exactly)	$2\cdot N\cdot H_{\mathrm{KV}}\cdot d_{h}$	$\frac{H_{\mathrm{KV}}}{H}\approx 6.25\%$
MLA	$N$ (exactly)	$N\cdot(d_{c}+d_{r})$	$\frac{d_{c}+d_{r}}{2Hd_{h}}\approx 1.76\%$
GDN	$N$ (ambiguous)	$H\cdot d_{h}^{2}$	$\frac{d_{h}}{2N}\approx 0\%$
SWA	$w$ (exactly)	$2\cdot w\cdot H_{\mathrm{KV}}\cdot d_{h}$	$\frac{w}{N}\cdot\frac{H_{\mathrm{KV}}}{H}\approx 0\%$
KSA	$N$ (summarized)	$2\cdot\frac{N}{k_{s}}\cdot H\cdot d_{h}$	$\frac{1}{k_{s}}\approx 12.5\%$

Table 1: KV-cache cost comparison for a sequence of

N

text tokens. Example settings use

k_{s}=8

k_{m}=128

H=128

d_{h}=128

H_{\mathrm{KV}}=8

d_{c}=512

, and

d_{r}=64

. Compression rates are relative to MHA.

Figure 6: KV cache size comparison across different mechanisms as sequence length grows. The inset uses a logarithmic vertical axis to separate the low-cache curves and labels C-KSA values at 128K.

Panel of Figure 7 — Figure 7: Kernel design for C-KSA and S-KSA: (a) direct KV addressing, (b) fused block scoring and selection under a $K$ -token budget, and (c) shared tiled prefill with KV reuse.

Configuration	1.9B	4B
Shared backbone
Number of layers	24	36
Hidden size	2048	2560
Intermediate size	6144	9728
Attention heads (Q/KV)	16/16	32/8
Head dimension	128	128

Table 2: Configurations of the selected 1.9B and 4B models. Module settings apply where the corresponding modules are used.

Figure 8: Sequential adaptation and parameter selection for KSA V2 at 1.9B.

Configuration	C-KSA		S-KSA
Configuration	Module-only	Full-parameter	Module-only
Trainable parameters	Aggregation modules	All	Aggregation and Indexer
Training tokens (B), 1.9B	6.7	3.4	18.5
Training tokens (B), 4B	6.7	15.9	4.2
Target learning rate	$10^{-4}$	$10^{-4}$	$10^{-3}$
LR warmup (updates)	200	400	200

Table 3: Adaptation hyperparameters and token budgets for the selected models.

Panel of Table 4 — Table 4: Long-context retrieval and general capabilities at 1.9B (%; pass@1 for code). Bold and underlined values mark the best and second-best scores, respectively, here and in subsequent tables.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial

Apr 27, 2026J/K move · Enter open · S save

Kwai Summary Attention Technical Report

Chenglong Chu, Guorui Zhou, Guowang Zhang, Han Li+34

Kuaishou

Self-Attention KV Caching Long-Context Language Model Inference Long-Context Language Modeling Transformer Attention Attention Mechanisms+1

Open paper AskPDF arXiv

Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agentic intelligence and recommendation system. However, the standard softmax attention exhibits quadratic time complexity with respect to sequence length. As the sequence length increases, this incurs substantial overhead in long-context settings, leading the training and inference costs of extremely long sequences deteriorate rapidly. Existing solutions mitigate this issue through two technique routings: i) Reducing the KV cache per layer, such as from the head-level compression GQA, and the embedding dimension-level compression MLA, but the KV cache remains linearly dependent on the sequence length at a 1:1 ratio. ii) Interleaving with KV Cache friendly architecture, such as local attention SWA, linear kernel GDN, but often involve trade-offs among KV Cache and long-context modeling effectiveness. Besides the two technique routings, we argue that there exists an intermediate path not well explored: {Maintaining a linear relationship between the KV cache and sequence length, but performing semantic-level compression through a specific ratio

k

}. This

O(n/k)

path does not pursue a ``minimum KV cache'', but rather trades acceptable memory costs for complete, referential, and interpretable retention of long distant dependency. Motivated by this, we propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts into learnable summary tokens.

Abstract

KV-Cache Compression

Explore similar work

Kwai Summary Attention Technical Report

SGD-KV: Summarization Guided KV Cache Compression

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression