CodeComp: Structural KV Cache Compression for Agentic Coding

Qiujiang Chen; Jing Xiong; Chenyang Zhao; Sidi Yang; Ngai Wong

cs.CLApr 11, 2026

CodeComp: Structural KV Cache Compression for Agentic Coding

Authors: Qiujiang Chen, Jing Xiong, Chenyang Zhao, Sidi Yang, Ngai Wong

Organizations: The University of Hong Kong · LMSYS Org

Abstract

Agentic code tasks such as fault localization and patch generation require processing long codebases under tight memory constraints, where the Key-Value (KV) cache becomes the primary inference bottleneck. Existing compression methods rely exclusively on attention signals to estimate token importance, systematically discarding structurally critical tokens such as call sites, branch conditions, and assignments that are essential for code understanding. We present CodeComp, a training-free KV cache compression framework that incorporates static program analysis into LLM inference via Code Property Graph priors extracted by Joern. Across bug localization and code generation benchmarks, CodeComp consistently outperforms attention-only compression baselines under equal memory budgets, recovering the majority of full-context accuracy under aggressive KV cache compression, while matching the patch generation quality of uncompressed full-context inference and integrating seamlessly into SGLang-based agentic coding pipelines without model modification.

Figures & tables

Figure 1: Attention-only compression discards structurally critical tokens such as function calls, branch predicates, and return statements. CodeComp uses CPG priors to explicitly preserve these tokens.

Panel of Figure 2 — Figure 2: Motivation analysis for KV cache compression in code. (a) Attention–structure mismatch: attention-based importance is weakly aligned with CPG-derived structural importance, causing structurally critical chunks to receive low attention scores. (b) Structural mis-pruning: under attention-only compression, a large fraction of critical structural tokens (e.g., callsite, assignment, and return) are incorrectly discarded. (c) Structure-aware retention: incorporating structural priors significantly improves the preservation of critical structural tokens across categories compared to attention-based methods.

Method	API Prec	API Rec	API Jac	Edit Dist
ParallelComp	0.1318	0.1536	0.0861	0.7046
SnapKV	0.0196	0.0909	0.0175	0.8615
CodeComp	0.1929	0.2975	0.1471	0.5998

Table 1: KV cache compression results on LCA.

Figure 3: Overview of CodeComp , a structure-aware KV cache compression framework. Given a query and retrieved repository context, we first select relevant chunks via PPL-based scoring (Step 1). We then extract structural anchors from static program analysis using Joern and CPG, identifying semantically critical code elements including function calls, control-flow predicates, return statements, and assignments (Step 2). These anchors are used to allocate chunk-level compression budgets and protect semantically critical spans (Step 3). Finally, attention-based compression fills the remaining capacity under these constraints (Step 4).

Model	Method	Cap.	GF Prec	GF Rec	GF F1	GF Jac	Patch Valid	Edit Dist
DS-Coder	No Comp.	1.0	0.338	0.338	0.338	0.338	0.875	0.740
DS-Coder	SnapKV	0.4	0.200	0.200	0.200	0.200	0.900	0.758
DS-Coder	ParallelComp	0.4	0.019	0.025	0.021	0.019	0.350	0.824
DS-Coder	CodeComp	0.4	0.250	0.250	0.250	0.250	1.000	0.743
DS-Coder	SnapKV	0.6	0.350	0.350	0.350	0.350	0.800	0.770
DS-Coder	ParallelComp	0.6	0.038	0.038	0.038	0.038	0.225	0.869

Table 2: Results on SWE-bench Lite. GF denotes ground-truth file-level evaluation. Cap. denotes the KV cache capacity ratio.

Model	Method	Cap.	InfBench	DebugBench	LongCodeQA	Str. Score	Avg.
Llama3-8B	No Comp.	1.0	0.20	0.74	0.59	1.00	0.51
Llama3-8B	SnapKV	0.4	0.10	0.41	0.48	0.36	0.33
Llama3-8B	ParallelComp	0.4	0.10	0.03	0.38	0.35	0.17
Llama3-8B	CodeComp	0.4	0.14	0.43	0.52	0.54	0.36
Llama3-8B	SnapKV	0.6	0.12	0.44	0.56	0.57	0.37
Llama3-8B	ParallelComp	0.6	0.11	0.17	0.31	0.58	0.20

Table 3: Bug localization accuracy on InfiniteBench-CodeDebug (InfBench), DebugBench, and LongCodeQA under different retention budgets. Str. Score denotes the average fraction of structurally critical tokens retained across layers and chunks, computed via CPG-derived span annotations. Best results under each compression ratio are bolded .

Cap	Method	0.9/1.1	0.75/1.25	0.5/1.5	0.25/1.75	0.1/1.9	Best
0.2	Capacity-only	0.433	0.400	0.417	0.400	0.450	0.450
	Span-only	–	–	–	–	–	0.617
	Capacity-Span	0.617	0.650	0.633	0.633	0.600	0.650
0.4	Capacity-only	0.283	0.250	0.233	0.267	0.250	0.283
	Span-only	–	–	–	–	–	0.633
	Capacity-Span	0.650	0.633	0.667	0.633	0.650	0.667

Table 4: Ablation results on DebugBench with 60 samples under different global compression ratios and budget-allocation strengths. Span-level preservation provides the main gains, while chunk-level budget allocation alone is weak. Capacity-Span benefits more under stronger allocation settings.

Panel of Figure 4 — Figure 4: Feature-level ablation and throughput analysis.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Cap.	API Prec	API Rec	API F1	API Jac	Edit Dist
DS-Coder	No Comp.	1.0	0.151	0.242	0.186	0.118	0.601
DS-Coder	ParallelComp	0.4	0.109	0.162	0.131	0.081	0.712
DS-Coder	SnapKV	0.4	0.048	0.365	0.085	0.044	0.716
DS-Coder	CodeComp	0.4	0.176	0.239	0.202	0.124	0.628
DS-Coder	ParallelComp	0.6	0.132	0.154	0.142	0.086	0.705
DS-Coder	SnapKV	0.6	0.050	0.408	0.088	0.047	0.722

Appendix

Table 5: Results on LCA CodeGen. API denotes API-level correctness. Cap. denotes the KV cache retention ratio.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 265 papers

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
Sep 25, 2026Zihan Wang, Cheng Tang, Lei Gong +4KV-Cache Management KV-Cache Compression
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China
SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
Feb 26, 2026Sanjay Kariyappa, G. Edward SuhKV-Cache Compression LLM Inference Acceleration
NVIDIA
AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding
Oct 13, 2026Xu Yang, Jiapeng Zhang, Zhiwei Zuo +5KV-Cache Compression Memory Compression
Hunan University · Central China Normal University · Tencent

Sep 25, 2026J/K move · Enter open · S save

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

Zihan Wang, Cheng Tang, Lei Gong, Chao Wang+3

University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

KV-Cache Management KV-Cache Compression LLM Inference Acceleration KV-Cache Pruning

Open paper AskPDF arXiv

Agentic LLM inference accumulates long KV caches across iterative observation-reasoning-action loops, imposing substantial memory overhead and limiting serving throughput. Existing compression methods emphasize overall output quality, overlooking the asymmetric importance of actions in driving task progress. Our key idea is to establish a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality. However, iterative execution, dynamic memory demands, and scattered action-critical entries pose challenges to eviction policies, budget allocation, and paged memory integration. To this end, we propose ActKV, the first KV cache compression framework tailored for agentic LLM inference. (i) Action-oriented KV cache eviction exploits stable action access patterns to retain entries critical to future actions, supporting reliable task progress under compression. (ii) Confidence-driven adaptive budget allocation uses LLM's intrinsic confidence to adapt the budget to evolving action-critical memory demands. (iii) Page-aware compression management standardizes compression into three primitives with customized kernels, realizing practical throughput gains. On long-trace tasks, ActKV retains an average of 98.53% of FullKV's accuracy with only 25.98% of its peak KV cache memory. It also achieves 3.97 times and 3.58 times FullKV's token and task throughput, delivering state-of-the-art performance.

Abstract

KV-Cache Compression

Explore similar work

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning

AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding