Long-horizon LLM agents play an important role in solving complex and sustained tasks, with ever-growing interaction contexts. However, these accumulated contexts will raise substantial computational cost and obscure key decision-relevant evidence. Existing methods typically train external compressors or context-management agents to rewrite context as shorter hard-token sequences, but such external selection may not align with the LLM's evolving representation and is necessarily lossy. In this paper, we introduce Agentic Soft-Token Compression (ASTC), a training-free compression method using soft tokens within LLM agent for long-horizon tasks. Specifically, ASTC uses the model's native recurrent dynamics to determine the already absorbed information, leveraging the commonly-used hybrid Gated Delta Network (GDN)-attention. It then carries this information forward as agentic soft tokens while retaining exact values as hard tokens. The resulting compact sequence reduces later-layer computation and yields reusable KV caches, extending the savings across the agent's trajectory. Extensive experiments on long-horizon tasks show that ASTC matches or exceeds the uncompressed agent's accuracy with up to 50.68% fewer end-to-end Attention Pairs, while trained compressors require 15.4 to 103.7 times more Attention Pairs per successful task. These findings highlight agentic soft tokens as a model-native way to carry absorbed context forward as the task representation evolves.
Figures & tables
Figure 1: Composition of accumulated context in real-world agent trajectories.
Figure 2: Overview of ASTC for in-model context compression and KV cache reuse.
AppWorld
OfficeBench
8-Objective QA
Method
Training-free
Acc.% ↑
D-KV% ↓
Attn.% ↓
Gen.% ↓
Acc.% ↑
D-KV% ↓
Attn.% ↓
Gen.% ↓
EM% ↑
F1% ↑
D-KV% ↓
Attn.% ↓
Gen.% ↓
Base
–
32.14
100.00
100.00
100.00
48.42
100.00
100.00
100.00
28.28
37.34
100.00
100.00
100.00
Hard-token compression
Prompting
✓
25.00
143.59
64.79
104.25
52.63
83.09
77.32
91.42
26.89
36.61
97.36
26.68
232.78
LLMLingua v1
✗
5.36
131.00
159.75
99.81
24.21
259.99
319.34
205.51
14.39
20.55
104.24
43.38
110.62
LLMLingua-2
✗
7.14
225.53
3,364.23
83.22
44.21
69.59
961.40
100.50
24.12
33.56
49.69
25.37
87.52
Table 1: Accuracy and end-to-end inference workload on three long-horizon agentic benchmarks.
Figure 3: Selector controls on AppWorld.
Figure 4: One-commit cache reuse and task outcomes. Middle: cumulative Attention Pairs saved since the commit; the observed task set varies across horizons.
Figure 5: Compression scopes on AppWorld. Bars: workload reduction; diamonds: accuracy; dashed: Base (32.14%). Workloads use each protocol’s matched Base. Tool is from Table 1 ; Assistant and Context are from Table 5 (Appendix E ).
Model
Method
Acc.% ↑
D-KV% ↓
Attn.% ↓
Gen.% ↓
Qwen3.5-9B
Base
32.14
100.00
100.00
100.00
ASTC
33.33
81.55
80.70
94.83
Qwen3.8-27B
Base
78.57
100.00
100.00
100.00
ASTC
77.98
91.70
64.43
103.82
Kimi-Linear-48B-A3B
Base
1.47
100.00
100.00
100.00
ASTC
4.41
84.01
88.76
94.70
Table 2: Cross-model transfer on AppWorld.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: ASTC compression in Venmo and Spotify agent trajectories.
Variant
Acc. ↑
D-KV ↓
Attn. ↓
Gen. ↓
Base
32.14%
100.00%
100.00%
100.00%
ASTC
33.33%
81.55%
80.70%
94.83%
No protection
20.83%
98.46%
68.63%
127.05%
No χ
32.14%
84.79%
78.11%
97.76%
No ψ
32.74%
89.72%
86.08%
99.65%
Hard deletion
32.74%
88.83%
86.85%
99.58%
Appendix
Table 3: Component ablations on AppWorld.
RULER 32K
LongBench v2
MBPP
Method
Acc.% ↑
D-KV% ↓
Attn.% ↓
Acc.% ↑
D-KV% ↓
Cache TB ↓
Pass% ↑
D-KV% ↓
Attn.% ↓
Base
97.173
100.00
100.00
40.95
100.00
2.065
84.78
100.00
100.00
ASTC
97.173
93.96
85.57
40.95
92.93
1.921
84.07
92.42
92.45
Appendix
Table 4: Additional-dataset results.
Figure 7: Task quality and inference resources across six datasets with frozen Qwen3.5-9B. Bars show accuracy, EM/F1, or pass rate on the left axis; lines show resource use relative to each dataset’s matched Base on the right axis (100%). Lower resource use is better. Panels (a)–(c) visualize Table 1 , and panels (d)–(f) complement Table 4 . LongBench v2 shows cache bytes because Attention Pairs were not reported. Supplementary generated-token and cache ratios use the underlying experiment records.
Scope
Acc.% ↑
D-KV% ↓
Attn.% ↓
Gen.% ↓
Base
32.14
100.00
100.00
100.00
Assistant
31.55
88.84
96.46
92.24
Context
30.95
79.50
80.32
90.81
Appendix
Table 5: Effect of compression scope on AppWorld.
Policy
Successes
Actions
Gen. tokens
Prefill
Decode
Total
Pairs/success
ASTC
56
4,203
291,482
103.771
23.297
127.068
2.269
Hidden-only
47
4,418
312,155
118.320
27.619
145.939
3.105
Random (101)
49
4,425
304,251
128.131
27.003
155.134
3.166
Appendix
Table 6: Full-split selector workload, including failed tasks. Pair counts are in billions ( 109 ). Pairs/success divides the total cost of all 168 tasks by the number of official all-requirements successes.
Control
Lower
Equal
Higher
Mean difference [95% CI]
Hidden-only
110
8
50
−112.32 [ −225.90 , −2.70 ]
Random (101)
112
8
48
−167.06 [ −294.27 , −41.46 ]
Appendix
Table 7: Task-paired Attention Pairs over all 168 tasks. Lower/equal/higher compares ASTC with the control. The difference is ASTC minus control, in millions ( 106 ) of pairs per task; brackets give a task-bootstrap 95% CI.
Policy
Logical pairs
Actual pairs
Total saved
Prefill saved
Decode saved
ASTC
152.197
127.068
16.51%
18.36%
7.16%
Hidden-only
164.365
145.939
11.21%
12.52%
5.10%
Random (101)
175.791
155.134
11.75%
13.06%
4.99%
Appendix
Table 8: Structural savings on each policy’s own inputs and actions. Logical counts assume no shortening; actual counts use realized layer lengths (billions of pairs). Savings are 1−∑Pactual/∑Plogical , with the analogous calculation for each component.
Metric
Task mean
95% CI
Action KL
5.86×10−4
[3.75,8.59]×10−4
Token top-1 agreement
99.719%
[99.576, 99.836]%
Reference-token Δ NLL
6.64×10−4
[1.68,13.00]×10−4
Appendix
Table 9: Overall fixed-input replay fidelity on all 35 triggered tasks. Intervals are task-bootstrap 95% CIs; actions within a task are not treated as independent samples. KL and Δ NLL are in nats/token.
Horizon
Mean KL
Top-1 agreement
Cumulative pairs saved
Saved fraction
0
4.26×10−3
99.412%
7.526
14.578%
1
6.20×10−4
99.529%
8.511
10.287%
2
4.40×10−4
99.633%
9.079
8.092%
4
1.87×10−4
100.000%
10.590
6.439%
8
1.30×10−4
100.000%
13.093
4.474%
16
6.80×10−5
100.000%
16.920
3.199%
Appendix
Table 10: Fixed-cohort persistence: the same 13 tasks at every horizon. Pairs saved are cumulative from the committing action, in millions; the last column averages each task’s cumulative saving fraction.
Audit
Verified count
Paired forwards with identical input-token digests
74,032/74,032
Equal cache lengths at attention layers 3 and 7
74,032/74,032
Exact R -position reduction at all six downstream attention layers
62,275/62,275
Pre-commit actions with exactly zero KL
267/267
No-event task actions with exactly zero KL
50/50
Triggered tasks with exactly one cache-local commit
35/35
Appendix
Table 11: Replay consistency audit. All counts include the full observed records; no action was truncated for scoring. A forward is one prefill or one single-token decode call.
Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-segment generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A12B) to achieve Pass@1 scores of 66.4% on SWE-bench Verified and 26.2% on Terminal-Bench 2.0, exceeding the base model under inference-time compaction by 6.6 and 4.9 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.7 points against the base model, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).
Yujiang Li, Zhenyu Hou, Yi Jing +2
Work done while YL, ZH, and YJ interned at Z.AI. · Tsinghua University
As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt. Furthermore, many methods require the input to fit within the target model's context window, and are generally incompatible with modern production inference engines. Encoder-decoder compressors, which map a long token sequence to a shorter sequence of latent embeddings consumed by a decoder, are an appealing alternative in principle. However, existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier. In this work, we revisit encoder-decoder compression and close this gap. We first perform an architecture search, pre-training many variants from scratch to determine how best to design and train encoder-decoder compressors. Guided by our findings, we continually pre-train a family of 0.6B-encoder, 4B-decoder models on nearly 400B tokens each, at compression ratios of 1:4, 1:8, and 1:16. We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage. LCLMs also unlock new efficient downstream applications: they serve as backbones for long-horizon agents that skim a compressed long context and adaptively expand relevant segments on demand, and they can compact the model's own generated reasoning into latent memory during inference, which is nearly lossless on the full MATH test set while being highly efficient, reducing peak KV usage by up to 7.9×.
Ang Li, Sean McLeish, Haozhe Chen +12
New York University · Modal Labs · Harvard University +5