cs.AIAug 28, 2026
SaveLoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
Organizations: DreamX Team, Alibaba Group · Beijing University of Posts and Telecommunications · UNSW Sydney; Data61, CSIRO
Abstract
Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the Controller: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the Worker, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I consists of 160 low-cost control questions; its execution cost is incurred once during benchmark construction, so each new Controller is evaluated through a single four-way choice with no code execution. Type II evaluates Controller guidance on 80 selected task slices, while Type III evaluates the corresponding 80 full tasks from their original states. On full tasks, the best observed Strict Success Rate is 36.25%, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages 60.6%, and Type II produces a positively correlated ordering under the main Core criterion (Spearman's rho = 0.7353). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena.
Figures & tables
Figure 1: The LoopArena harness. Both conditions start from the same restored task state and use the same Worker, coding tools, and task evaluator. In the Controller-guided condition, a temporary Reporter agent summarizes the Worker’s progress after each round, and the harness deterministically formats the report as an Evidence Packet . The Controller reads this structured, read-only summary and issues a Loop Contract specifying the next Worker segment, or decides to stop. In the no-control condition, the Worker runs without Controller guidance.
Figure 2: Constructing the three LoopArena settings. Type III evaluates a full coding task from its original state. Type II starts from a prepared intermediate workspace and evaluates one slice of the same task. Type I uses a restorable control point from a Controller-guided trajectory and determines the correct option by replaying all four candidate Loop Contracts.
| Instances | Per-run statistics | |||
|---|---|---|---|---|
| Setting | Total | By source | Worker turns | Control cycles |
| Type I | 160 | 40 / 53 / 67 | — | — |
| Type II | 80 | 21 / 29 / 30 | 86.61–145.30 | 3.55–5.42 |
| Type III | 80 | 21 / 29 / 30 | 189.41–355.94 | 8.16–13.45 |
Table 1: Benchmark composition and evaluation statistics. Type II and Type III use the same 80 source tasks. Source counts are listed in SCBench/BeyondSWE/LoopsBench order. Bar lengths represent the average Worker turns per run across the evaluated Controllers. The ranges in the final two columns span the corresponding per-Controller means. Type I requires no Worker execution.
| Type I | Type II (task slice) | Type III (full task) | |||
| Method | Contract Acc. (%) | SSR (%) | Est. cost (\downarrow$ | SSR (%) | Est. cost (\downarrow$ |
| Reference policies | |||||
| No control | — | 49.17 | 1.62 | 27.50 | 2.61 |
| Fixed control | — | 55.00 | 5.62 | 31.25 | 9.47 |
| Controller models | |||||
| Qwen3.6-35B-A3B | 43.125 | 55.83 | 4.21 | 33.75 | 11.58 |
Table 2: Main results. Type I reports Contract Accuracy; Type II and Type III report Strict Success Rate and mean estimated inference cost per run. The best Controller result in each column is bold and the second-best distinct value is underlined; reference policies are not ranked. Resource details appear in Appendix 14 . A dash marks an inapplicable field.
| Controller | SSR (%) | Worker turns | Controller cost ($/run) | Total cost ($/run) |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | 33.75 | 224.36 | 0.20 | 11.58 |
| Qwen3.7-Plus | 34.17 | 243.33 | 0.23 | 12.69 |
| DeepSeek-V4-Flash-0731 | 28.33 | 330.58 | 0.15 | 32.66 |
| GLM 5.2 | 24.17 | 189.41 | 0.81 | 11.97 |
| GPT-5.5 | 36.25 | 355.94 | 6.18 | 34.68 |
| Claude Opus 4.8 | 33.75 | 324.09 | 7.16 | 20.63 |
Table 3: Success and execution cost on full tasks. SSR uses Core. Worker turns and costs are means over all 240 runs per Controller. Total estimated inference cost includes Worker, Reporter, and Controller calls under the main table’s no-cache price schedule.
| Candidate | Next instruction | Label |
|---|---|---|
| A | Remove a dependency required by a newly supported Django configuration. | Incorrect |
| B | Remove a dependency still used by the active latest-version environment. | Incorrect |
| C | Keep the unused drf39 dependency and inspect the matrix again. | Incorrect |
| D | Remove the orphaned drf39 dependency while preserving active configurations. | Correct |
Table 4: Shortened Type I example. The question asks which instruction should be issued next; the tested Controller does not execute code.
| Setting | Check | Requirement |
|---|---|---|
| All | task binding | task text, starting workspace, and evaluator refer to the same frozen source instance |
| All | solution leakage | model-visible inputs exclude official solution code, private scoring output, and later trajectory events |
| Type I | answer stability | the two predeclared replay schedules identify the same unique winner |
| Type I | post-freeze audit | reviewers assess clarity, candidate plausibility, and leakage without changing the replay-defined answer |
| Type II | task-slice validity | the prepared starting workspace fails at least one selected-stage check, while the source-provided completed state passes all requirements through that stage |
| Type II/III | evaluator validity | the frozen evaluator distinguishes a completed source task from an incomplete state |
Table 5: Benchmark construction checks and audits. Automated checks enforce the execution protocol; post-freeze review characterizes Type I quality without changing its replay-defined answer.
| Controller | Type I inference interface |
|---|---|
| Qwen3.6-35B-A3B | API; temperature 0; maximum 20,480 output tokens |
| Qwen3.7-Plus | API; temperature 0; maximum 20,480 output tokens |
| DeepSeek-V4-Flash-0731 | API; temperature 0; maximum 20,480 output tokens |
| GLM 5.2 | API; temperature 0; maximum 20,480 output tokens |
| GPT-5.5 | API ( gpt-5.5-0424-global ); provider-default thinking; no temperature or seed; maximum 20,480 output tokens |
| Claude Opus 4.8 | API ( claude-opus-4-8 ); provider-default thinking; no temperature or seed; maximum 20,480 output tokens |
Table 6: Type I inference interfaces. Exact provider identifiers and request records are preserved with the run manifest.
| Quantity | Value or reporting rule |
|---|---|
| Type I panel | 160 questions; one response per Controller and question |
| Type II panel | 80 task slices paired with Type III |
| Type III panel | 80 unique full tasks |
| Executable repeats | separate runs for every Controller, reference policy, and task |
| Reference reuse | one no-control set and one fixed-control set per task and Worker configuration, shared across Controllers |
| Primary executable score | Strict Success Rate |
Table 7: Evaluation plan. Type II and Type III use the same Controllers, repeat count, and paired full tasks.
| Controller | Correct | Accuracy (%) | Invalid (%) | Est. cost ($/160 questions) |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | 69/160 | 43.125 | 0.000 | 1.57 |
| Qwen3.7-Plus | 97/160 | 60.625 | 0.000 | 1.45 |
| DeepSeek-V4-Flash-0731 | 73/160 | 45.625 | 0.625 | 0.56 |
| GLM 5.2 | 87/160 | 54.375 | 1.875 | 6.81 |
| GPT-5.5 | 99/160 | 61.875 | 0.000 | 21.67 |
| Claude Opus 4.8 | 90/160 | 56.250 | 0.000 | 30.56 |
Table 8: Type I results. Estimated cost is the standardized marginal no-cache cost of 160 Controller responses and excludes the one-time candidate-execution cost incurred during benchmark construction.
| Controller | SCBench (40) | BeyondSWE (53) | LoopsBench (67) |
|---|---|---|---|
| Qwen3.6-35B-A3B | 18/40 (45.00) | 32/53 (60.38) | 19/67 (28.36) |
| Qwen3.7-Plus | 33/40 (82.50) | 41/53 (77.36) | 23/67 (34.33) |
| DeepSeek-V4-Flash-0731 | 21/40 (52.50) | 32/53 (60.38) | 20/67 (29.85) |
| GLM 5.2 | 29/40 (72.50) | 37/53 (69.81) | 21/67 (31.34) |
| GPT-5.5 | 35/40 (87.50) | 46/53 (86.79) | 18/67 (26.87) |
| Claude Opus 4.8 | 29/40 (72.50) | 40/53 (75.47) | 21/67 (31.34) |
Table 9: Type I accuracy by source benchmark. Parentheses show percentages.
| Model | Input tokens per call | Input price ($/1M tokens) | Output price ($/1M tokens) |
|---|---|---|---|
| Qwen3.6-35B-A3B | all supported | 0.2652 | 1.5911 |
| Qwen3.7-Plus | up to 256K | 0.2946 | 1.1786 |
| 256K–1M | 0.8839 | 3.5358 | |
| DeepSeek-V4-Flash-0731 | all supported | 0.1473 | 0.2946 |
| GLM 5.2 | all supported | 1.1786 | 4.1250 |
| GPT-5.5 | up to 272K | 5.0000 | 30.0000 |
Table 10: Frozen price schedule for inference-cost estimation. Prices were retrieved on August 16, 2026 from the providers’ public price schedules ( Alibaba Cloud, 2026 ; OpenAI, 2026 ; Anthropic, 2026 ) . Qwen list prices are the China (Beijing) rates in CNY, converted at the August 14, 2026 official central parity rate of CNY 6.7878 per USD ( State Administration of Foreign Exchange, 2026 ) . We use public list prices and do not apply temporary promotions, caching, batch, priority, regional, or enterprise discounts.
| Estimated cost ($/run) | Execution length | |||||
| Worker | Reporter | Controller | Total | Worker turns | Control cycles | |
| Type II (task slice) | ||||||
| No control | 1.62 | — | — | 1.62 | 68.50 | — |
| Fixed control | 5.62 | — | — | 5.62 | 105.82 | 2.85 |
| Qwen3.6-35B-A3B | 3.86 | 0.26 | 0.10 | 4.21 | 103.37 | 4.24 |
| Qwen3.7-Plus | 5.62 | 0.31 | 0.10 | 6.04 | 114.56 | 4.15 |
Table 11: Mean estimated no-cache inference cost and execution length. All reported entries are averaged over 240 scored runs per policy or Controller and setting. Worker turns count inner-loop ReAct turns; control cycles count outer-loop handoffs. A dash marks a model-call role or execution-length metric that does not apply to the policy. Fixed control uses neither a Reporter nor a model Controller. Totals may differ from displayed component sums because of rounding.
| Type II | Type III | |||
|---|---|---|---|---|
| Controller | Qwen | DeepSeek | Qwen | DeepSeek |
| Qwen3.6-35B-A3B | 36/60 | 34/60 | 28/60 | 20/60 |
| Qwen3.7-Plus | 33/60 | 36/60 | 26/60 | 23/60 |
| DeepSeek-V4-Flash-0731 | 26/60 | 27/60 | 19/60 | 15/60 |
| GLM 5.2 | 29/60 | 31/60 | 19/60 | 12/60 |
Table 12: Worker-side comparison on 20 source tasks. Each entry is Core successes out of 60 runs (20 tasks with three repeats). Qwen denotes a Qwen3.7-Plus Worker and Reporter; DeepSeek denotes a DeepSeek-V4-Flash-0731 Worker and Reporter. The Controller named in each row is held fixed across the two Worker-side conditions.
| Analysis | Information retained | Accuracy (%) |
|---|---|---|
| Uniform random | none | 25.00 |
| Majority position | answer position only | 30.625 |
| Prefer advance | action labels only | 24.271 |
| Prefer verify | action labels only | 28.490 |
| Prefer stop | action labels only | 22.813 |
| Shortest candidate | serialized candidate length | 23.125 |
Table 13: Type I shortcut analyses. Tested baselines remain below the weakest tested Controller (43.125%). Action-preference rules choose uniformly among candidates with the preferred action, or among all four when that action is absent.
| Method | Type II tasks 0/3, 1/3, 2/3, 3/3 | Type III tasks 0/3, 1/3, 2/3, 3/3 |
|---|---|---|
| No control | 29, 10, 15, 26 | 51, 6, 9, 14 |
| Fixed control | 27, 6, 15, 32 | 51, 3, 6, 20 |
| Qwen3.6-35B-A3B | 26, 7, 14, 33 | 45, 8, 8, 19 |
| Qwen3.7-Plus | 28, 5, 10, 37 | 46, 9, 2, 23 |
| DeepSeek-V4-Flash-0731 | 29, 10, 11, 30 | 48, 10, 8, 14 |
| GLM 5.2 | 30, 18, 17, 15 | 53, 5, 13, 9 |
Table 14: Run-to-run outcome stability. Entries count tasks with exactly 0, 1, 2, or 3 successful runs among the three recorded repeats.
| Method | Type II SCBench | Type II BeyondSWE | Type II LoopsBench | Type III SCBench | Type III BeyondSWE | Type III LoopsBench |
|---|---|---|---|---|---|---|
| No control | 34/63 | 39/87 | 45/90 | 5/63 | 21/87 | 40/90 |
| Fixed control | 43/63 | 37/87 | 52/90 | 3/63 | 19/87 | 53/90 |
| Qwen3.6-35B-A3B | 38/63 | 40/87 | 56/90 | 5/63 | 20/87 | 56/90 |
| Qwen3.7-Plus | 38/63 | 39/87 | 59/90 | 6/63 | 19/87 | 57/90 |
| DeepSeek-V4-Flash-0731 | 36/63 | 36/87 | 50/90 | 6/63 | 19/87 | 43/90 |
| GLM 5.2 | 18/63 | 36/87 | 43/90 | 0/63 | 17/87 | 41/90 |
Table 15: Executable success by source benchmark. Entries are successful runs over the 63 SCBench, 87 BeyondSWE, or 90 LoopsBench runs in each method and setting.
| Evaluation setting | Model output | Execution | Evaluated object |
|---|---|---|---|
| Final-state coding benchmark | patch or repository state | evaluated coding stack | end-to-end coding ability |
| Interactive agent benchmark | action trajectory | evaluated agent stack | agent plus scaffold |
| Process or critique benchmark | score, label, or critique | usually no fixed downstream executor | local step quality |
| Loop-system benchmark | loop or harness configuration | coding model inside each submitted system | complete loop system |
| LoopArena Type I | one candidate instruction | no Worker execution during evaluation | single control decision |
| LoopArena Type II | repeated Loop Contracts | fixed Worker on a selected task slice | runtime loop control on a selected task slice |
Table 16: Comparison by evaluated object. The three LoopArena settings use different amounts of execution but all target the Controller’s loop-control ability.
| Component | Prompt composition |
|---|---|
| Worker | The shared Worker system prompt, the task instruction, and either the bootstrap instruction or a Controller-derived continuation. |
| Reporter | The Reporter system prompt and a reporting request containing the task instruction and quoted Worker conversation. |
| Controller | The Controller system prompt followed by successive rendered Evidence Packets in one persistent Controller conversation. |
| Type I | The Controller decision policy with a selection-specific instruction, followed by one rendered Packet, four candidate decisions, and the answer instruction. |
| Reference policies | No control uses the shared Worker prompt with an autonomous-start instruction. Fixed control uses the shared Worker prompt and a deterministic continuation at each handoff; it invokes neither a Reporter nor a model Controller. |
Table 17: Prompt composition in LoopArena. Benchmark instances provide the task, current state, reported evidence, and candidate decisions; the following sections give the fixed prompts and templates.
More in this topic
LLM Agent Orchestration
Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.
See all 463 papers