Latent-Condensed Transformer for Efficient Long Context Modeling

Zeng You; Yaofo Chen; Qiuwu Chen; Ying Sun; Shuhai Zhang; Yingjian Li; Yaowei Wang; Mingkui Tan

cs.CLApr 14, 2026

Latent-Condensed Transformer for Efficient Long Context Modeling

Authors: Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan

Organizations: South China University of Technology · Pengcheng Laboratory · AIGCode · Harbin Institute of Technology · Pazhou Laboratory

Abstract

Large language models (LLMs) face significant challenges in processing long contexts due to the linear growth of the key-value (KV) cache and quadratic complexity of self-attention. Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation. However, sparse methods cannot operate natively on MLA's compressed latent structure, missing opportunities for joint optimization. In this paper, we propose Latent-Condensed Attention (LCA), which directly condenses context within MLA's latent space, where the representation is disentangled into semantic latent vectors and positional keys. LCA separately aggregates semantic vectors via query-aware pooling and preserves positional keys via anchor selection. This approach jointly reduces both computational cost and KV cache without adding parameters. Beyond MLA, LCA's design is architecture-agnostic and readily extends to other attention mechanisms such as GQA. Theoretically, we prove a length-independent error bound. Experiments show LCA achieves up to 2.5 $\times$ prefilling speedup and 90% KV cache reduction at 128K context while maintaining competitive performance.

Figures & tables

Figure 1: An overview of Latent-Condensed Attention (LCA). The history context is condensed into a compact set of representatives via group-wise condensation in the latent space: semantic information is aggregated through weighted pooling of $\mathbf{C}^{KV}$ , while positional information is preserved via anchor selection from $\mathbf{K}^{R}$ . The recent context is retained in full fidelity without condensation.

Methods	Computation	Memory	No Extra Params
Minference	$<\mathcal{O}(L^{2})$	$\mathcal{O}(L)$	$\checkmark$
FlexPrefill	$<\mathcal{O}(L^{2})$	$\mathcal{O}(L)$	$\checkmark$
KDA	$<\mathcal{O}(L^{2})$	$\mathcal{O}(L)$
Ours	$\mathcal{O}(Lm)$	$\mathcal{O}(m)$	$\checkmark$

Table 1: Comparison of efficient attention methods.

Methods	S. QA	M. QA	Sum.	F. S.	Syn.	Code	Avg.↑	FTL (s) ↓
DeepSeek V2-Lite (MLA)	23.92	11.15	17.52	63.19	3.09	58.17	29.51	3.20
$\bullet$ MInference Jiang et al. (2024)	14.54	5.61	13.80	41.15	2.17	41.01	19.71	2.99 (1.1 $\times$ )
$\bullet$ FlexPrefill Lai et al. (2025)	14.35	5.94	14.10	48.00	2.22	41.72	21.05	2.51 (1.3 $\times$ )
$\bullet$ KDA Kimi et al. (2025b)	18.81	11.68	14.22	56.97	3.25	57.97	27.15	2.47 (1.3 $\times$ )
$\bullet$ LCA (Ours)	22.61	8.90	23.74	58.27	2.33	58.67	29.09	1.80 (1.8 $\times$ )

Table 2: Comparisons on LongBench-E Bai et al. (2023) . We report the latency of DeepSeek V2-Lite within contexts of 64K on H200 GPUs.

Methods	4K	8K	16K	32K	64K	128K	Avg.↑	FTL (s) ↓
DeepSeek V2-Lite ( MLA )	79.51	78.82	62.28	58.33	47.57	23.96	58.91	10.78
$\bullet$ MInference Jiang et al. (2024)	76.88	69.69	44.06	20.31	10.31	4.34	37.60	5.66 (1.9 $\times$ )
$\bullet$ FlexPrefill Lai et al. (2025)	72.81	70.00	56.25	19.06	9.38	7.19	39.11	5.38 (2.0 $\times$ )
$\bullet$ KDA Kimi et al. (2025b)	72.22	73.61	64.24	51.39	44.10	22.22	54.63	4.96 (2.2 $\times$ )
$\bullet$ LCA (Ours)	77.19	77.19	63.75	59.69	50.63	24.38	58.80	4.40 (2.5 $\times$ )

Table 3: Comparisons on RULER Hsieh et al. (2024) across 4-128K context. We report the latency of DeepSeek V2-Lite within contexts of 128K on H200 GPUs.

Figure 2: Efficiency comparison of different context lengths. (a) presents the first-token generation latency. (b) shows the per-token (interval) latency during decoding. (c) illustrates the GPU memory footprint of KV cache.

Table 6

Table 7

#Queries	S. QA	M. QA	Sum.	F. S.	Syn.	Code	Avg.
32	23.01	8.82	23.87	58.33	2.41	59.09	29.26
16 (default)	22.61	8.90	23.74	58.27	2.33	58.67	29.09
8	22.34	8.37	23.17	56.22	2.09	58.41	28.43

Table 8: Ablation on the number of queries used to compute importance scores. Default setting is 16.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Context Length	4K	32K	128K
$\\|\mathbf{k}_{i}\\|_{2}$	20.00	22.13	21.39
$\delta_{k}$	0.43	0.64	0.77
$\delta_{k}/\\|\mathbf{k}_{i}\\|_{2}$	2.15%	2.89%	3.60%
$\\|\mathbf{v}_{i}\\|_{2}$	20.87	21.38	21.23
$\delta_{v}$	0.68	0.73	0.77
$\delta_{v}/\\|\mathbf{v}_{i}\\|_{2}$	3.26%	3.41%	3.63%

Appendix

Table 9: Statistics across different context lengths.

Metric	Multi-Doc. QA	Summarization	Code
$\\|\mathbf{k}_{i}\\|_{2}$	22.76	21.42	20.07
$\delta_{k}$	0.67	0.98	0.89
$\delta_{k}/\\|\mathbf{k}_{i}\\|_{2}$	2.94%	4.58%	4.43%
$\\|\mathbf{v}_{i}\\|_{2}$	19.09	24.78	29.78
$\delta_{v}$	0.56	0.51	0.98
$\delta_{v}/\\|\mathbf{v}_{i}\\|_{2}$	2.93%	2.06%	3.29%

Appendix

Table 10: Statistics across different task types.

$g$	LongBench Avg.	RULER Avg. (4K–128K)	RULER@128K
32	28.32	58.18	23.96
16	29.09	58.80	24.38
8	29.38	59.42	24.63

Appendix

Table 11: Effect of group size

g

with fixed window size

w=1024

$w$	LongBench Avg.	RULER Avg. (4K–128K)	RULER@128K
512	28.84	57.14	23.68
1024	29.09	58.80	24.38
2048	29.14	59.52	25.53

Appendix

Table 12: Effect of window size

w

with fixed group size

g=16

Explore similar work

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
Jun 8, 2026Zhanchao Xu, Haoyang Li, Qingfa Xiao +4Long-Context Language Model Inference LLM Inference Acceleration
Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
CompLLM: Compression for Long Context Q&A
Sep 23, 2025Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1LLM Compression Long-Context Language Model Inference
Amazon · Center For Research in Computer Vision, University of Central Florida

Jun 8, 2026J/K move · Enter open · S save

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

Zhanchao Xu, Haoyang Li, Qingfa Xiao, Fei Teng+3

Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST

Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention KV-Cache Compression

Open paper AskPDF arXiv

Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts. We observe two distinct entropy patterns among attention heads: Rigid Heads, whose entropy stays near zero across input segments, and Dynamic Heads, whose entropy fluctuates significantly. Crucially, the distribution of these types is context-dependent and cannot be predetermined offline. We therefore propose EntropyInfer, a training-free framework that uses attention entropy to adaptively allocate compute at the granularity of individual heads and segments during prefilling. For decoding, we introduce a latent KV cache compression scheme that leverages generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Extensive experiments on Llama, Qwen and openPangu model series show that EntropyInfer consistently outperforms baselines including SnapKV, AdaKV, and CritiPrefill, achieving up to 2.39

\times

end-to-end speedup beyond 100k tokens with minimal quality degradation compared to full attention. The code is released in https://github.com/SHA-4096/EntropyInfer.

Abstract

Explore similar work

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

SGD-KV: Summarization Guided KV Cache Compression

CompLLM: Compression for Long Context Q&A