cs.AINov 19, 2026

CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks

Authors: Jiayi Chen, Lizhe Chen, Yujia Chen, Shihan Dou, Ming Zhang, Chendi Ge, Shiguang Ni, Tao Gui, 

Organizations: Fudan University · Hunyuan Team, Tencent · SIGS, Tsinghua University

Abstract

As language-model agents are applied to increasingly extended tasks, their working context continuously accumulates interaction history, tool observations, intermediate decisions, and task progress. Context compaction offers a practical way to reduce this growing history while allowing execution to continue, yet the compaction ability of modern LLMs remains insufficiently understood. We introduce CompAct-Bench, a benchmark for evaluating working-context compaction through its downstream effect on agent task completion. Starting from successful agent trajectories, we insert compaction boundaries at different stages of execution, compress the preceding working context with the model under evaluation, and let the same executor continue the original task. To enable fair comparison across compactors, we evaluate them under controlled retention ratios using a lightweight iterative ratio-adjustment procedure. CompAct-Bench covers information-seeking, software-engineering, and workspace-oriented tasks through BrowseComp, SWE-bench, and an internal CompanyBench evaluation. Our experiments show that working-context compaction remains highly lossy even for frontier models: Claude- and GPT-family models perform strongest overall, yet substantial degradation in downstream task accuracy persists under aggressive compaction. Further analysis reveals that failures are often caused not only by omitted information, but also by distorted task state, such as turning unresolved issues into apparent conclusions or forgetting previously established results, which can lead to incorrect decisions or redundant execution. These findings highlight working-context compaction as a distinct and still challenging ability for LLMs operating in extended agent tasks.

Figures & tables

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Benchmarks Are Surprisingly Compaction-Friendly

    Oct 22, 2026Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2Context Compaction for LLM AgentsLLM Context Compression

  2. CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

    Jul 6, 2026Yujiang Li, Zhenyu Hou, Yi Jing +2Software Engineering AgentsLong-Horizon LLM Agents

  3. Lost in Context Compaction: When LLM Systems Drop Side Constraints

    Jul 31, 2026Zhiqi Wang, Yichi Zhang, Dongwon Lee +2Persistent Memory for Language ModelsInstruction Following