Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Organizations: University of Notre Dame · IBM Research · MIT-IBM Watson AI Lab · Carnegie Mellon University ·
Abstract
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.
Figures & tables
| Dataset | Sampled | Solved | Reported | Minutes | Tokens | ||||||
| Synthetic | |||||||||||
| TMax | 100 | 94 | 16.3 | 100% | 0% | 1 | 5 | 58 | 5.8 | 0.9 | 6.6 |
| FACET | 210 | 53 | 11.86 | 92% | 8% | 2 | 7 | 204 | 8.2 | 2.1 | 19.3 |
| SETA | 100 | 91 | – | 98% | 2% | 1 | 5 | 25 | 7.0 | 0.8 | 8.4 |
| RST | 200 | 58 | – | 79% | 21% | 2 | 7 | 119 | 8.2 | 1.2 | 11.0 |
| Human-curated | |||||||||||
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Harness | Time limit | Output | Wait per command | Turns |
| Synthetic | |||||
| TMax | mini-SWE-agent based | – | 10,000 characters | Blocks up to 120 s | 64 |
| FACET | terminus-2 | 2 min | 10,000 bytes | Up to 60 s, not blocking | No limit |
| SETA | CAMEL | 48 min | 1,000 characters | Blocks up to 20 s, then polls | 30 |
| RST | terminus-2 | 18 min | 10,000 bytes | Up to 60 s, not blocking | – |
| Human-curated | |||||
| Dataset | Groups | Evidence that a model passed | Sampled |
| Synthetic | |||
| TMax | Three complexity levels, each split into three groups by the fewest turns of a passing Gemini run; the hardest level as one group | Gemini-3-Flash passes at least 2 of 8 runs | 100 |
| FACET | Three groups by number of checks | None released | 210 |
| SETA | Synth: hard level, drawn at random; Evolve: three groups by the fewest iterations of a passing Qwen3-30B run | Evolve: Qwen3-30B passes at least 2 of 8 runs in the authors’ intermediate data; Synth: none | 100 |
| RST | Four groups by lines of the reference solution | Qwen3.5-27B runs, matched only to lineages (the tasks derived from one seed); lineages with no passed task are excluded | 200 |
| Human-curated | |||
| Dataset | Labels | Sample against original | : sample reweighted |
| Synthetic | |||
| TMax | Four complexity levels | First three about 30% each (original 25%); hardest 9% (original 26%) | 0% 0% |
| FACET | None (all “medium”) | Three groups by number of checks, close to the original | 7.5% (not reweighted) |
| SETA | Synth and Evolve, three levels each | Synth hard 52% (original 14%); Synth medium (original 57%) not sampled | 2.2% 1.5% † |
| RST | Seed difficulty | Medium 27% (original 18%) | 20.7% 14.9% |
| Human-curated | |||
| Solved tasks | All sampled tasks | |||||
| Dataset | Minutes | Tokens | Turns | Minutes | Tokens | |
| Synthetic | ||||||
| TMax | 5.8 | 0.9 | 6.6 | 6.4 | 1.0 | 6.9 |
| FACET | 8.2 | 2.1 | 19.3 | 10.1 | 3.3 | 24.2 |
| SETA | 7.0 | 0.8 | 8.4 | 8.3 | 1.0 | 10.1 |
| RST | 8.2 | 1.2 | 11.0 | 12.0 | 2.1 | 15.2 |
| Function | Passes when |
| expect_ok NAME – CMD | CMD exits with status 0 |
| expect_out NAME PATTERN – CMD | a line of the output of CMD matches PATTERN, whatever the exit status |
| expect_no_out NAME PATTERN – CMD | CMD exits with status 0 and no line of its output matches PATTERN |
| expect_fail NAME PATTERN – CMD | CMD fails and its output matches PATTERN, for an access that must be refused |
| expect_eq NAME ACTUAL EXPECTED [KIND] | the two values are equal as KIND: text (default), int, num, version, bool, set, seq or exact |
| expect_file NAME PATH | PATH is a non-empty file or a directory |
| Change | Checks expected to fail | Holds | Share |
| Checks of segment run before its work, on | 601 / 602 | 99.8% | |
| Previous segment skipped: run without | 462 / 501 | 92.2% | |
| One event of segment removed from | 158 / 165 | 95.8% | |
| Key step of segment removed: run instead of | 598 / 604 | 99.0% | |
| Key step of segment removed, then run | 421 / 500 | 84.2% |
| Per segment | Per task | |
| Lines of the reference solution | 52 [18, 130] | 330 [134, 626] |
| Words of the instruction | 234 [171, 336] | 1,406 [1,153, 1,790] |
| Names stated in the instruction | 14 [6, 24] | 83 [50, 123] |
| Segment | The instruction asks to | Key step |
| 1 | Generate the build system of the source checkout and configure it for a given installation layout | The NetworkManager pkg-config module is absent; the configure script actually needs libnm, which is installed |
| 2 | Build and stage the installation under /app/nm-l2tp-stage, leaving /usr untouched | The NetworkManager plugin libraries are staged under the multiarch library directory, not next to the service executable |
| 3 | Make the staged pppd plugin loadable by the installed pppd 2.4.9 while keeping the requested 2.4.7 layout, and write a manifest of the staged plugins | Copying the plugin into the 2.4.9 directory succeeds, but pppd looks up a bare plugin name in its own plugin directory; only loading it by its staged path shows that it works |
| 4 | Package the staged tree into one archive that can be unpacked elsewhere | The staged files record absolute paths of the stage, so a plain archive unpacks without error but works elsewhere only after these paths are made relative |
| 5 | Build a local Debian package from the archived payload | dpkg-deb rejects the tree until it has a control file, valid installation paths and executable permissions |
| 6 | Relocate the deployment to /opt/nm-l2tp, keeping all earlier results | The relocated service file still points to paths under /usr |