cs.AINov 1, 2026

Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks

Authors: Yu Jiang, Yue Huang, Hang Hua, Yuyang Wu, Zixuan Weng, Olexandr Isayev, Yue Zhao, Zinan Lin, 

Organizations: University of Notre Dame · IBM Research · MIT-IBM Watson AI Lab · Carnegie Mellon University · 

Abstract

Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Recursive Synthesis for Long-Horizon Terminal Tasks

    Aug 5, 2026Zhongzhi Li, Yucheng Shi, Zongxia Li +11Terminal AgentsSynthetic Benchmark Generation

  2. Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training

    Jul 30, 2026Zhihong Pan, Jiyuan He, Kai Zhang +5Synthetic Data GenerationTerminal Agents

  3. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

    Jul 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +12Terminal-Agent BenchmarksLong-Horizon Agent Evaluation