Tools Need Not Speak in Text: Latent Context Compression for Agents

Yuetong Liu; Zhehao Zhou; Yitong Zhang; Haorui Chen; Jia Li

cs.AINov 4, 2026

Tools Need Not Speak in Text: Latent Context Compression for Agents

Authors: Yuetong Liu, Zhehao Zhou, Yitong Zhang, Haorui Chen, Jia Li

Organizations: Tsinghua AI Agent Group, Tsinghua University

Abstract

Large language model (LLM) agents typically append raw tool outputs to their context. Over multi-step interactions, long and noisy observations incur substantial context overhead and dilute the agent's attention to task-relevant information. Existing token pruning and summarization methods remain constrained by discrete text representations, limiting the achievable compression ratios. In this work, we challenge the prevailing assumption that agents must consume tool observations as explicit text tokens. To this end, we propose Coca-Cola, a continuous latent context compression model for LLM agents. Coca-Cola compresses tool observations into latent tokens and embeds them into the input embedding space of the agent. We further introduce Action-Preserving Training, which aligns the agent's next-action distributions under full and compressed observations. This trains the compression model to learn action-sufficient latent tokens that retain decision-critical information. Experiments with web-search agents on four QA benchmarks and two model families show that Coca-Cola reduces the agent's token consumption by 70.01%-89.24% and end-to-end runtime by 24.45%-55.97%, while consistently improving task performance. These results suggest that Coca-Cola offers a promising approach for context compression in agents. Our code is available at https://github.com/THU-Agent/Coca-cola.

Figures & tables

Figure 1: Tool observation redundancy in LLM agents. Left: Tool returns increasingly dominate the agent context, accounting for 96.7% of total tokens by Step 4, while individual observations exhibit substantial length. Right: A representative web-search example illustrating that only a few fragments of a lengthy tool output are relevant to the agent’s next decision.

Figure 2: Overview of Coca-Cola. Left: The latent tool bottleneck encodes raw tool observations into compact continuous latent representations, conditioned on the task instruction and current reasoning state. Middle: Action-preserving distillation aligns the next-action distributions of a full-observation teacher and a latent-conditioned student, updating only the compressor while keeping the agent backbone frozen. Right: During inference, raw tool observations are replaced by latent representations after each tool call, enabling efficient multi-step agent interactions.

	NQ					TriviaQA
	Vanilla	LongLLM Lingua	LLM Summary	ICAE	Coca-Cola	Vanilla	LongLLM Lingua	LLM Summary	ICAE	Coca-Cola
$I_{\mathrm{token}}\,\downarrow$	14362.12	8607.04	6099.83	5926.23	1766.53 [1.5pt] $\downarrow$ 87.7%	10022.93	5445.57	3791.53	3038.35	1413.12 [1.5pt] $\downarrow$ 85.9%
$O_{\mathrm{token}}\,\downarrow$	992.69	1232.61	1080.51	665.57	417.69 [1.5pt] $\downarrow$ 57.9%	779.34	1048.02	679.36	535.42	349.36 [1.5pt] $\downarrow$ 55.2%
$\mathrm{Comp\_rate}\,\uparrow$	–	35.92%	53.24%	57.07%	85.78%	–	39.89%	58.61%	66.92%	83.68%
Time (s) $\downarrow$	96.19	109.48	121.02	87.06	64.64 [1.5pt] $\downarrow$ 32.8%	73.30	86.91	91.67	64.89	55.38 [1.5pt] $\downarrow$ 24.4%
	2Wiki					HotpotQA

Table 1: Efficiency comparison across four web-search QA datasets using Qwen3.5-35B-A3B and Qwen3.5-4B. Best results are in bold. Blue annotations under Coca-Cola indicate reductions (%) relative to Vanilla in token usage and runtime.

	NQ					TriviaQA
	Vanilla	LongLLM Lingua	LLM Summary	ICAE	Coca-Cola	Vanilla	LongLLM Lingua	LLM Summary	ICAE	Coca-Cola
$I_{\mathrm{token}}\,\downarrow$	17388.56	5779.61	3903.19	5068.65	4265.42 [1.5pt] $\downarrow$ 75.5%	11176.91	4689.20	2842.51	3510.28	2595.00 [1.5pt] $\downarrow$ 76.8%
$O_{\mathrm{token}}\,\downarrow$	887.74	797.81	734.77	805.93	536.37 [1.5pt] $\downarrow$ 39.6%	738.96	688.27	634.84	617.94	413.28 [1.5pt] $\downarrow$ 44.1%
$\mathrm{Comp\_rate}\,\uparrow$	–	64.01%	74.62%	67.86%	73.73%	–	54.87%	70.82%	65.36%	74.75%
Time (s) $\downarrow$	89.29	91.06	106.47	83.63	42.17 [1.5pt] $\downarrow$ 52.8%	91.36	89.93	128.36	72.33	41.97 [1.5pt] $\downarrow$ 54.1%
	2Wiki					HotpotQA

Table 2: Efficiency comparison across four web-search QA datasets using GLM-4.7-Flash and GLM-Z1-9B. Best results are in bold. Blue annotations under Coca-Cola indicate reductions (%) relative to Vanilla in token usage and runtime.

Method	Single-hop QA				Multi-hop QA
	NQ		TriviaQA		2Wiki		HotpotQA
	EM	F1	EM	F1	EM	F1	EM	F1
Qwen3.5-35B-A3B & Qwen3.5-4B
Vanilla Model	23.67	33.17	56.10	62.87	17.30	26.94	11.00	16.41
LongLLMLingua	19.00	25.81	44.30	48.69	21.30	34.25	11.90	19.75
LLMSummary	24.65	32.35	54.70	61.38	21.06	31.77	23.10	29.60

Table 3: Web search performance comparison. Best results within each model setting are in bold. Arrows in the Coca-Cola rows indicate relative changes (%) over the Vanilla Model for each metric.

Figure 3: Effect of the number of latent tokens

m

on NQ (GLM-4.7-Flash & GLM-Z1-9B). F1 improves steadily as the latent budget increases.

Method	EM (%)	Compression rate (%)
NQ
Full	23.50	73.73
w/o Reasoning State	21.45	80.43
w/o Task Goal	20.02	78.38
Observation-only	18.34	79.94

Table 4: Context-component ablation on NQ (GLM-4.7-Flash & GLM-Z1-9B,

m=128

Figure 4: Case study on HotpotQA #377 comparing the Vanilla Model and Coca-Cola.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Web-search reasoning prompt used in our experiments.

Context Compaction for LLM Agents

See all 50 papers

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM Agents Context Compression
The University of Texas at Austin
FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents
Sep 29, 2026Shantanu Dixit, Anson Bastos, Xuchao Zhang +2Long-Horizon LLM Agents LLM Context Management
M365 Research, Microsoft
In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents
Oct 30, 2026Xuning He, Kai Fan, Runyu Zhang +4Context Compaction for LLM Agents LLM Context Compression
Shanghai Jiao Tong University · Nanyang Technological University · Nankai University +2

Sep 26, 2026J/K move · Enter open · S save

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

Ritul Satish, Prasoon Sinha, Akiho Kawada, Neeraja J. Yadwadkar

The University of Texas at Austin

Context Compaction for LLM Agents Context Compression LLM Agent Harnesses LLM Agent Evaluation

Open paper AskPDF arXiv

As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.

Abstract

Context Compaction for LLM Agents

Explore similar work

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents

In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents