In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents

Xuning He; Kai Fan; Runyu Zhang; Binghuan Wang; Yuxiang Zhang; Xue Jiang; Yihong Dong

cs.CLOct 30, 2026

In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents

Authors: Xuning He, Kai Fan, Runyu Zhang, Binghuan Wang, Yuxiang Zhang, Xue Jiang, Yihong Dong

Organizations: Shanghai Jiao Tong University · Nanyang Technological University · Nankai University · Tongji University ·

Abstract

Long-horizon LLM agents play an important role in solving complex and sustained tasks, with ever-growing interaction contexts. However, these accumulated contexts will raise substantial computational cost and obscure key decision-relevant evidence. Existing methods typically train external compressors or context-management agents to rewrite context as shorter hard-token sequences, but such external selection may not align with the LLM's evolving representation and is necessarily lossy. In this paper, we introduce Agentic Soft-Token Compression (ASTC), a training-free compression method using soft tokens within LLM agent for long-horizon tasks. Specifically, ASTC uses the model's native recurrent dynamics to determine the already absorbed information, leveraging the commonly-used hybrid Gated Delta Network (GDN)-attention. It then carries this information forward as agentic soft tokens while retaining exact values as hard tokens. The resulting compact sequence reduces later-layer computation and yields reusable KV caches, extending the savings across the agent's trajectory. Extensive experiments on long-horizon tasks show that ASTC matches or exceeds the uncompressed agent's accuracy with up to 50.68% fewer end-to-end Attention Pairs, while trained compressors require 15.4 to 103.7 times more Attention Pairs per successful task. These findings highlight agentic soft tokens as a model-native way to carry absorbed context forward as the task representation evolves.

Figures & tables

Figure 1: Composition of accumulated context in real-world agent trajectories.

Figure 2: Overview of ASTC for in-model context compression and KV cache reuse.

Hard-token compression
		AppWorld				OfficeBench				8-Objective QA
Method	Training-free	Acc.% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Gen.% $\downarrow$	Acc.% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Gen.% $\downarrow$	EM% $\uparrow$	F1% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Gen.% $\downarrow$
Base	–	32.14	100.00	100.00	100.00	48.42	100.00	100.00	100.00	28.28	37.34	100.00	100.00	100.00
Prompting	✓	25.00	143.59	64.79	104.25	52.63	83.09	77.32	91.42	26.89	36.61	97.36	26.68	232.78
LLMLingua v1	✗	5.36	131.00	159.75	99.81	24.21	259.99	319.34	205.51	14.39	20.55	104.24	43.38	110.62
LLMLingua-2	✗	7.14	225.53	3,364.23	83.22	44.21	69.59	961.40	100.50	24.12	33.56	49.69	25.37	87.52

Table 1: Accuracy and end-to-end inference workload on three long-horizon agentic benchmarks.

Figure 3: Selector controls on AppWorld.

Figure 4: One-commit cache reuse and task outcomes. Middle: cumulative Attention Pairs saved since the commit; the observed task set varies across horizons.

Figure 5: Compression scopes on AppWorld. Bars: workload reduction; diamonds: accuracy; dashed: Base (32.14%). Workloads use each protocol’s matched Base. Tool is from Table 1 ; Assistant and Context are from Table 5 (Appendix E ).

Model	Method	Acc.% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Gen.% $\downarrow$
Qwen3.5-9B	Base	32.14	100.00	100.00	100.00
Qwen3.5-9B	ASTC	33.33	81.55	80.70	94.83
Qwen3.8-27B	Base	78.57	100.00	100.00	100.00
Qwen3.8-27B	ASTC	77.98	91.70	64.43	103.82
Kimi-Linear-48B-A3B	Base	1.47	100.00	100.00	100.00
Kimi-Linear-48B-A3B	ASTC	4.41	84.01	88.76	94.70

Table 2: Cross-model transfer on AppWorld.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: ASTC compression in Venmo and Spotify agent trajectories.

Variant	Acc. $\uparrow$	D-KV $\downarrow$	Attn. $\downarrow$	Gen. $\downarrow$
Base	32.14%	100.00%	100.00%	100.00%
ASTC	33.33%	81.55%	80.70%	94.83%
No protection	20.83%	98.46%	68.63%	127.05%
No $\chi$	32.14%	84.79%	78.11%	97.76%
No $\psi$	32.74%	89.72%	86.08%	99.65%
Hard deletion	32.74%	88.83%	86.85%	99.58%

Appendix

Table 3: Component ablations on AppWorld.

	RULER 32K			LongBench v2			MBPP
Method	Acc.% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Acc.% $\uparrow$	D-KV% $\downarrow$	Cache TB $\downarrow$	Pass% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$
Base	97.173	100.00	100.00	40.95	100.00	2.065	84.78	100.00	100.00
ASTC	97.173	93.96	85.57	40.95	92.93	1.921	84.07	92.42	92.45

Appendix

Table 4: Additional-dataset results.

Figure 7: Task quality and inference resources across six datasets with frozen Qwen3.5-9B. Bars show accuracy, EM/F1, or pass rate on the left axis; lines show resource use relative to each dataset’s matched Base on the right axis (100%). Lower resource use is better. Panels (a)–(c) visualize Table 1 , and panels (d)–(f) complement Table 4 . LongBench v2 shows cache bytes because Attention Pairs were not reported. Supplementary generated-token and cache ratios use the underlying experiment records.

Scope	Acc.% $\uparrow$	D-KV% $\downarrow$	Attn.% $\downarrow$	Gen.% $\downarrow$
Base	32.14	100.00	100.00	100.00
Assistant	31.55	88.84	96.46	92.24
Context	30.95	79.50	80.32	90.81

Appendix

Table 5: Effect of compression scope on AppWorld.

Policy	Successes	Actions	Gen. tokens	Prefill	Decode	Total	Pairs/success
ASTC	56	4,203	291,482	103.771	23.297	127.068	2.269
Hidden-only	47	4,418	312,155	118.320	27.619	145.939	3.105
Random (101)	49	4,425	304,251	128.131	27.003	155.134	3.166

Appendix

Table 6: Full-split selector workload, including failed tasks. Pair counts are in billions (

10^{9}

). Pairs/success divides the total cost of all 168 tasks by the number of official all-requirements successes.

Control	Lower	Equal	Higher	Mean difference [95% CI]
Hidden-only	110	8	50	$-112.32$ [ $-225.90$ , $-2.70$ ]
Random (101)	112	8	48	$-167.06$ [ $-294.27$ , $-41.46$ ]

Appendix

Table 7: Task-paired Attention Pairs over all 168 tasks. Lower/equal/higher compares ASTC with the control. The difference is ASTC minus control, in millions (

10^{6}

) of pairs per task; brackets give a task-bootstrap 95% CI.

Policy	Logical pairs	Actual pairs	Total saved	Prefill saved	Decode saved
ASTC	152.197	127.068	16.51%	18.36%	7.16%
Hidden-only	164.365	145.939	11.21%	12.52%	5.10%
Random (101)	175.791	155.134	11.75%	13.06%	4.99%

Appendix

Table 8: Structural savings on each policy’s own inputs and actions. Logical counts assume no shortening; actual counts use realized layer lengths (billions of pairs). Savings are

1-\sum P_{\mathrm{actual}}/\sum P_{\mathrm{logical}}

, with the analogous calculation for each component.

Metric	Task mean	95% CI
Action KL	$5.86\times 10^{-4}$	$[3.75,8.59]\times 10^{-4}$
Token top-1 agreement	99.719%	[99.576, 99.836]%
Reference-token $\Delta$ NLL	$6.64\times 10^{-4}$	$[1.68,13.00]\times 10^{-4}$

Appendix

Table 9: Overall fixed-input replay fidelity on all 35 triggered tasks. Intervals are task-bootstrap 95% CIs; actions within a task are not treated as independent samples. KL and

\Delta

NLL are in nats/token.

Horizon	Mean KL	Top-1 agreement	Cumulative pairs saved	Saved fraction
0	$4.26\times 10^{-3}$	99.412%	7.526	14.578%
1	$6.20\times 10^{-4}$	99.529%	8.511	10.287%
2	$4.40\times 10^{-4}$	99.633%	9.079	8.092%
4	$1.87\times 10^{-4}$	100.000%	10.590	6.439%
8	$1.30\times 10^{-4}$	100.000%	13.093	4.474%
16	$6.80\times 10^{-5}$	100.000%	16.920	3.199%

Appendix

Table 10: Fixed-cohort persistence: the same 13 tasks at every horizon. Pairs saved are cumulative from the committing action, in millions; the last column averages each task’s cumulative saving fraction.

Audit	Verified count
Paired forwards with identical input-token digests	74,032/74,032
Equal cache lengths at attention layers 3 and 7	74,032/74,032
Exact $R$ -position reduction at all six downstream attention layers	62,275/62,275
Pre-commit actions with exactly zero KL	267/267
No-event task actions with exactly zero KL	50/50
Triggered tasks with exactly one cache-local commit	35/35

Appendix

Table 11: Replay consistency audit. All counts include the full observed records; no action was truncated for scoring. A forward is one prefill or one single-token decode call.

Context Compaction for LLM Agents

See all 43 papers

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
Jul 6, 2026Yujiang Li, Zhenyu Hou, Yi Jing +2Software Engineering Agents Long-Horizon LLM Agents
Work done while YL, ZH, and YJ interned at Z.AI. · Tsinghua University
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM Agents Context Compression
The University of Texas at Austin
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · Modal Labs · Harvard University +5

Jul 6, 2026J/K move · Enter open · S save

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang+1

Work done while YL, ZH, and YJ interned at Z.AI. · Tsinghua University

Software Engineering Agents Long-Horizon LLM Agents RL for LLM Agents Context Compression Context Compaction for LLM Agents

Open paper AskPDF arXiv

Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-segment generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A12B) to achieve Pass@1 scores of 66.4% on SWE-bench Verified and 26.2% on Terminal-Bench 2.0, exceeding the base model under inference-time compaction by 6.6 and 4.9 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.7 points against the base model, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).

Abstract

Context Compaction for LLM Agents

Explore similar work

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

End-to-End Context Compression at Scale