SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification

Yulu An; Yifei Feng; Xiangyang Yin

cs.CLNov 4, 2026

SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification

Authors: Yulu An, Yifei Feng, Xiangyang Yin

Organizations: New York University

Abstract

LLM agents use context compaction to manage growing interaction histories, but summarizing long histories with large models can introduce substantial delays. Compaction therefore requires balancing information retention against latency. We observe that structured summary slots can retain information when generated independently. Based on this observation, we present SpecCompact, a framework that accelerates compaction through slot-parallel semantic speculation. SpecCompact fine-tunes a small drafter and generates slots in parallel. The target model verifies each proposed line and repairs rejected content, while subsequent drafting uses the updated slot prefix. System optimizations further reduce repeated context processing and model calls. Across AppWorld and $τ^3$ benchmarks with two target models, SpecCompact achieves $2.04$ -- $5.63\times$ compaction speedups over Hermes-style summarization. Separate downstream evaluations reveal model- and task-dependent performance trade-offs. A user study within Hermes further shows an $8.68\times$ median compaction speedup.

Figures & tables

Figure 1: Task success and compaction speedup with Gemma4-31B-IT, measured in separate evaluations.

Figure 2: Slot structure and independent generation. (a) Fields in OpenClaw and Hermes compaction templates. (b) Reference adequacy over 100 trajectories on LiveClawBench ( Long et al., 2026 ) , with six slots per trajectory.

Figure 3: Overview of SpecCompact.

Figure 4: Execution optimizations in SpecCompact: independent slot scheduling, shared-prefix caching, draft buffering, line-aware stopping, and lightweight checks with termination rules.

Models	Methods	AppWorld Normal		AppWorld Challenge		$\tau^{3}$
Models	Methods	TGC $\uparrow$	SGC $\uparrow$	TGC $\uparrow$	SGC $\uparrow$	Success $\uparrow$
Gemma4- 31B-IT	Full History	62.50 [51.79,72.62]	46.43 [33.93,58.93]	53.24 [45.80,60.43]	41.73 [33.81,49.64]	52.08 [45.00,58.75]
	OpenClaw	58.93 [47.62,69.64]	44.64 [32.14,57.14]	46.52 [39.09,53.72]	33.09 [25.18,41.01]	47.50 [39.17,55.42]
	Hermes	57.74 [47.02,67.86]	39.29 [26.79,51.79]	47.24 [40.05,54.44]	32.37 [25.18,40.29]	50.00 [42.08,57.92]
	SpecCompact	54.76 [44.05,65.48]	37.50 [25.00,50.00]	43.17 [35.97,50.12]	28.06 [20.86,35.25]	47.50 [40.00,55.00]
Qwen3.5- 122B-A10B	Full History	40.48 [29.76,51.19]	26.79 [16.07,37.50]	20.62 [15.35,26.14]	8.63 [4.32,13.67]	67.92 [63.33,72.08]

Table 1: Agent task performance (%). Full History is a reference without compaction. Bold marks the best compaction result. Gray subscripts show 95% confidence intervals.

Models	Methods	LongBench		RULER
Models	Methods	GovReport $\uparrow$	QMSum $\uparrow$	8k $\uparrow$	16k $\uparrow$
Gemma4- 31B-IT	Full History	28.36 [27.73,29.02]	24.34 [23.28,25.41]	91.72 [88.77,94.62]	92.12 [89.22,95.02]
	Target-self	25.05 [24.51,25.59]	18.35 [17.61,19.10]	71.00 [67.68,74.67]	71.02 [67.72,74.70]
	SpecCompact	22.28 [21.10,23.37]	17.52 [16.51,18.52]	66.50 [63.67,69.70]	68.55 [65.32,71.87]
Qwen3.5- 122B-A10B	Full History	31.16 [30.53,31.82]	22.98 [22.03,23.96]	90.92 [87.63,93.82]	90.92 [87.63,94.17]
	Target-self	28.25 [27.71,28.79]	18.54 [17.86,19.25]	72.98 [69.33,76.67]	72.15 [68.87,75.80]

Table 2: Long-context performance on LongBench (ROUGE-L) and RULER at 8k and 16k (official matching scores), scaled to 100. Execution failures are scored as zero. Bold marks the best compaction result.

Models	Methods	AppWorld				$\tau^{3}$		LongBench 16k
		Normal		Challenge		$\tau^{3}$		LongBench 16k
		Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$
Gemma4- 31B-IT	Hermes	38.89	1.00 $\times$	39.12	1.00 $\times$	36.21	1.00 $\times$	74.13	1.00 $\times$
	OpenClaw	16.44	2.37 $\times$	18.70	2.09 $\times$	13.19	2.75 $\times$	63.55	1.17 $\times$
	SpecCompact	8.26	4.71 $\times$	9.14	4.28 $\times$	6.43	5.63 $\times$	29.72	2.49 $\times$
Qwen3.5- 122B-A10B	Hermes	18.03	1.00 $\times$	16.98	1.00 $\times$	17.81	1.00 $\times$	29.43	1.00 $\times$

Table 3: Compaction latency and speedup on 20 matched inputs per workload. Speedup is relative to Hermes. Bold marks the best result within each target. SpecCompact uses an additional drafter GPU.

Benchmark	Before $\downarrow$	After $\downarrow$	Speedup $\uparrow$
Gemma4-31B-IT
AppWorld Normal	20.17	8.26	2.44 $\times$
AppWorld Challenge	21.56	9.14	2.36 $\times$
$\tau^{3}$	12.07	6.43	1.88 $\times$
Qwen3.5-122B-A10B
AppWorld Normal	9.55	7.49	1.28 $\times$

Table 4: Combined effect of system optimizations on 20 matched inputs per workload. Both versions use slot-parallel execution. Times are in seconds.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: The drafter prompt template and the contract for each slot.

Panel of Figure 6 — Figure 6: Target verification prompt (top) and the message structure with a shared prefix (bottom).

Context Compaction for LLM Agents

See all 50 papers

Parallel Context Compaction for Long-Horizon LLM Agent Serving
May 22, 2026Musa Cim, Burak Topcu, Chita Das +1LLM Inference Efficiency LLM Agent Workflow Optimization
The Pennsylvania State University
Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents
May 9, 2026Zhuofu Chen, Rui Pan, Yinwei Dai +1LLM Agent Memory Long-Horizon LLM Agents
Princeton University
Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents
Oct 14, 2026Guanghui Min, Chen ChenContext Compaction for LLM Agents LLM Agent Reliability
Department of Computer Science, University of Virginia

May 22, 2026J/K move · Enter open · S save

Parallel Context Compaction for Long-Horizon LLM Agent Serving

Musa Cim, Burak Topcu, Chita Das, Mahmut Taylan Kandemir

The Pennsylvania State University

LLM Inference Efficiency LLM Agent Workflow Optimization Long-Horizon LLM Agents LLM Context Management Context Compaction for LLM Agents

Open paper AskPDF arXiv

Long-horizon LLM agents accumulate growing conversation histories that eventually exceed the model's context window. Context compaction via LLM-based summarization keeps the conversation bounded, but summarization is inherently lossy and the blocking call stalls agent inference for tens of seconds. Moreover, the operator has no fine-grained control over summary volume since prompt instructions are largely ignored, and as context grows, both the amount of output tokens the model produces and the information it retains fluctuate substantially from run to run, making the agent's retained knowledge unpredictable across runs. We introduce parallel compaction for long-horizon agentic flows and characterize it against the sequential synchronous baseline across four backbones spanning 8B to 120B parameters, mixing dense and MoE architectures with reasoning and non-reasoning models, on the HotpotQA multi-hop QA and LoCoMo long-context dialogue benchmarks. Parallel compaction gives the operator fine-grained, predictable control over summary volume and enables more targeted prompt engineering per block. At matched compaction decode volume, it reduces end-to-end wall time and improves compaction throughput over the sequential baseline.

Abstract

Context Compaction for LLM Agents

Explore similar work

Parallel Context Compaction for Long-Horizon LLM Agent Serving

Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents

Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents