LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

Yi Wang; Haopeng Zhang; Chengxiang Huang; Rui Dai; Kaikui Liu; Piotr Koniusz; Xiangxiang Chu

cs.AIAug 28, 2026

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

Authors: Yi Wang, Haopeng Zhang, Chengxiang Huang, Rui Dai, Kaikui Liu, Piotr Koniusz, Xiangxiang Chu

Organizations: DreamX Team, Alibaba Group · Beijing University of Posts and Telecommunications · UNSW Sydney; Data61, CSIRO

Abstract

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the Controller: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the Worker, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I consists of 160 low-cost control questions; its execution cost is incurred once during benchmark construction, so each new Controller is evaluated through a single four-way choice with no code execution. Type II evaluates Controller guidance on 80 selected task slices, while Type III evaluates the corresponding 80 full tasks from their original states. On full tasks, the best observed Strict Success Rate is 36.25%, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages 60.6%, and Type II produces a positively correlated ordering under the main Core criterion (Spearman's rho = 0.7353). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena.

Figures & tables

Figure 1: The LoopArena harness. Both conditions start from the same restored task state and use the same Worker, coding tools, and task evaluator. In the Controller-guided condition, a temporary Reporter agent summarizes the Worker’s progress after each round, and the harness deterministically formats the report as an Evidence Packet . The Controller reads this structured, read-only summary and issues a Loop Contract specifying the next Worker segment, or decides to stop. In the no-control condition, the Worker runs without Controller guidance.

Figure 2: Constructing the three LoopArena settings. Type III evaluates a full coding task from its original state. Type II starts from a prepared intermediate workspace and evaluates one slice of the same task. Type I uses a restorable control point from a Controller-guided trajectory and determines the correct option by replaying all four candidate Loop Contracts.

	Instances		Per-run statistics
Setting	Total	By source	Worker turns	Control cycles
Type I	160	40 / 53 / 67	—	—
Type II	80	21 / 29 / 30	86.61–145.30	3.55–5.42
Type III	80	21 / 29 / 30	189.41–355.94	8.16–13.45

Table 1: Benchmark composition and evaluation statistics. Type II and Type III use the same 80 source tasks. Source counts are listed in SCBench/BeyondSWE/LoopsBench order. Bar lengths represent the average Worker turns per run across the evaluated Controllers. The ranges in the final two columns span the corresponding per-Controller means. Type I requires no Worker execution.

Reference policies
	Type I	Type II (task slice)		Type III (full task)
Method	Contract Acc. (%) $\uparrow$	SSR (%) $\uparrow$	Est. cost ( $/run)$ \downarrow$	SSR (%) $\uparrow$	Est. cost ( $/run)$ \downarrow$
No control	—	49.17	1.62	27.50	2.61
Fixed control	—	55.00	5.62	31.25	9.47
Controller models
Qwen3.6-35B-A3B	43.125	55.83	4.21	33.75	11.58

Table 2: Main results. Type I reports Contract Accuracy; Type II and Type III report Strict Success Rate and mean estimated inference cost per run. The best Controller result in each column is bold and the second-best distinct value is underlined; reference policies are not ranked. Resource details appear in Appendix 14 . A dash marks an inapplicable field.

Controller	SSR (%)	Worker turns	Controller cost ($/run)	Total cost ($/run)
Qwen3.6-35B-A3B	33.75	224.36	0.20	11.58
Qwen3.7-Plus	34.17	243.33	0.23	12.69
DeepSeek-V4-Flash-0731	28.33	330.58	0.15	32.66
GLM 5.2	24.17	189.41	0.81	11.97
GPT-5.5	36.25	355.94	6.18	34.68
Claude Opus 4.8	33.75	324.09	7.16	20.63

Table 3: Success and execution cost on full tasks. SSR uses Core. Worker turns and costs are means over all 240 runs per Controller. Total estimated inference cost includes Worker, Reporter, and Controller calls under the main table’s no-cache price schedule.

Candidate	Next instruction	Label
A	Remove a dependency required by a newly supported Django configuration.	Incorrect
B	Remove a dependency still used by the active latest-version environment.	Incorrect
C	Keep the unused drf39 dependency and inspect the matrix again.	Incorrect
D	Remove the orphaned drf39 dependency while preserving active configurations.	Correct

Table 4: Shortened Type I example. The question asks which instruction should be issued next; the tested Controller does not execute code.

Setting	Check	Requirement
All	task binding	task text, starting workspace, and evaluator refer to the same frozen source instance
All	solution leakage	model-visible inputs exclude official solution code, private scoring output, and later trajectory events
Type I	answer stability	the two predeclared replay schedules identify the same unique winner
Type I	post-freeze audit	reviewers assess clarity, candidate plausibility, and leakage without changing the replay-defined answer
Type II	task-slice validity	the prepared starting workspace fails at least one selected-stage check, while the source-provided completed state passes all requirements through that stage
Type II/III	evaluator validity	the frozen evaluator distinguishes a completed source task from an incomplete state

Table 5: Benchmark construction checks and audits. Automated checks enforce the execution protocol; post-freeze review characterizes Type I quality without changing its replay-defined answer.

Controller	Type I inference interface
Qwen3.6-35B-A3B	API; temperature 0; maximum 20,480 output tokens
Qwen3.7-Plus	API; temperature 0; maximum 20,480 output tokens
DeepSeek-V4-Flash-0731	API; temperature 0; maximum 20,480 output tokens
GLM 5.2	API; temperature 0; maximum 20,480 output tokens
GPT-5.5	API ( gpt-5.5-0424-global ); provider-default thinking; no temperature or seed; maximum 20,480 output tokens
Claude Opus 4.8	API ( claude-opus-4-8 ); provider-default thinking; no temperature or seed; maximum 20,480 output tokens

Table 6: Type I inference interfaces. Exact provider identifiers and request records are preserved with the run manifest.

Quantity	Value or reporting rule
Type I panel	160 questions; one response per Controller and question
Type II panel	80 task slices paired with Type III
Type III panel	80 unique full tasks
Executable repeats	$K=3$ separate runs for every Controller, reference policy, and task
Reference reuse	one no-control set and one fixed-control set per task and Worker configuration, shared across Controllers
Primary executable score	Strict Success Rate

Table 7: Evaluation plan. Type II and Type III use the same Controllers, repeat count, and paired full tasks.

Controller	Correct	Accuracy (%)	Invalid (%)	Est. cost ($/160 questions)
Qwen3.6-35B-A3B	69/160	43.125	0.000	1.57
Qwen3.7-Plus	97/160	60.625	0.000	1.45
DeepSeek-V4-Flash-0731	73/160	45.625	0.625	0.56
GLM 5.2	87/160	54.375	1.875	6.81
GPT-5.5	99/160	61.875	0.000	21.67
Claude Opus 4.8	90/160	56.250	0.000	30.56

Table 8: Type I results. Estimated cost is the standardized marginal no-cache cost of 160 Controller responses and excludes the one-time candidate-execution cost incurred during benchmark construction.

Controller	SCBench (40)	BeyondSWE (53)	LoopsBench (67)
Qwen3.6-35B-A3B	18/40 (45.00)	32/53 (60.38)	19/67 (28.36)
Qwen3.7-Plus	33/40 (82.50)	41/53 (77.36)	23/67 (34.33)
DeepSeek-V4-Flash-0731	21/40 (52.50)	32/53 (60.38)	20/67 (29.85)
GLM 5.2	29/40 (72.50)	37/53 (69.81)	21/67 (31.34)
GPT-5.5	35/40 (87.50)	46/53 (86.79)	18/67 (26.87)
Claude Opus 4.8	29/40 (72.50)	40/53 (75.47)	21/67 (31.34)

Table 9: Type I accuracy by source benchmark. Parentheses show percentages.

Model	Input tokens per call	Input price ($/1M tokens)	Output price ($/1M tokens)
Qwen3.6-35B-A3B	all supported	0.2652	1.5911
Qwen3.7-Plus	up to 256K	0.2946	1.1786
	256K–1M	0.8839	3.5358
DeepSeek-V4-Flash-0731	all supported	0.1473	0.2946
GLM 5.2	all supported	1.1786	4.1250
GPT-5.5	up to 272K	5.0000	30.0000

Table 10: Frozen price schedule for inference-cost estimation. Prices were retrieved on August 16, 2026 from the providers’ public price schedules ( Alibaba Cloud, 2026 ; OpenAI, 2026 ; Anthropic, 2026 ) . Qwen list prices are the China (Beijing) rates in CNY, converted at the August 14, 2026 official central parity rate of CNY 6.7878 per USD ( State Administration of Foreign Exchange, 2026 ) . We use public list prices and do not apply temporary promotions, caching, batch, priority, regional, or enterprise discounts.

Type II (task slice)
	Estimated cost ($/run)				Execution length
	Worker	Reporter	Controller	Total	Worker turns	Control cycles
No control	1.62	—	—	1.62	68.50	—
Fixed control	5.62	—	—	5.62	105.82	2.85
Qwen3.6-35B-A3B	3.86	0.26	0.10	4.21	103.37	4.24
Qwen3.7-Plus	5.62	0.31	0.10	6.04	114.56	4.15

Table 11: Mean estimated no-cache inference cost and execution length. All reported entries are averaged over 240 scored runs per policy or Controller and setting. Worker turns count inner-loop ReAct turns; control cycles count outer-loop handoffs. A dash marks a model-call role or execution-length metric that does not apply to the policy. Fixed control uses neither a Reporter nor a model Controller. Totals may differ from displayed component sums because of rounding.

	Type II		Type III
Controller	Qwen	DeepSeek	Qwen	DeepSeek
Qwen3.6-35B-A3B	36/60	34/60	28/60	20/60
Qwen3.7-Plus	33/60	36/60	26/60	23/60
DeepSeek-V4-Flash-0731	26/60	27/60	19/60	15/60
GLM 5.2	29/60	31/60	19/60	12/60

Table 12: Worker-side comparison on 20 source tasks. Each entry is Core successes out of 60 runs (20 tasks with three repeats). Qwen denotes a Qwen3.7-Plus Worker and Reporter; DeepSeek denotes a DeepSeek-V4-Flash-0731 Worker and Reporter. The Controller named in each row is held fixed across the two Worker-side conditions.

Analysis	Information retained	Accuracy (%)
Uniform random	none	25.00
Majority position	answer position only	30.625
Prefer advance	action labels only	24.271
Prefer verify	action labels only	28.490
Prefer stop	action labels only	22.813
Shortest candidate	serialized candidate length	23.125

Table 13: Type I shortcut analyses. Tested baselines remain below the weakest tested Controller (43.125%). Action-preference rules choose uniformly among candidates with the preferred action, or among all four when that action is absent.

Method	Type II tasks 0/3, 1/3, 2/3, 3/3	Type III tasks 0/3, 1/3, 2/3, 3/3
No control	29, 10, 15, 26	51, 6, 9, 14
Fixed control	27, 6, 15, 32	51, 3, 6, 20
Qwen3.6-35B-A3B	26, 7, 14, 33	45, 8, 8, 19
Qwen3.7-Plus	28, 5, 10, 37	46, 9, 2, 23
DeepSeek-V4-Flash-0731	29, 10, 11, 30	48, 10, 8, 14
GLM 5.2	30, 18, 17, 15	53, 5, 13, 9

Table 14: Run-to-run outcome stability. Entries count tasks with exactly 0, 1, 2, or 3 successful runs among the three recorded repeats.

Method	Type II SCBench	Type II BeyondSWE	Type II LoopsBench	Type III SCBench	Type III BeyondSWE	Type III LoopsBench
No control	34/63	39/87	45/90	5/63	21/87	40/90
Fixed control	43/63	37/87	52/90	3/63	19/87	53/90
Qwen3.6-35B-A3B	38/63	40/87	56/90	5/63	20/87	56/90
Qwen3.7-Plus	38/63	39/87	59/90	6/63	19/87	57/90
DeepSeek-V4-Flash-0731	36/63	36/87	50/90	6/63	19/87	43/90
GLM 5.2	18/63	36/87	43/90	0/63	17/87	41/90

Table 15: Executable success by source benchmark. Entries are successful runs over the 63 SCBench, 87 BeyondSWE, or 90 LoopsBench runs in each method and setting.

Evaluation setting	Model output	Execution	Evaluated object
Final-state coding benchmark	patch or repository state	evaluated coding stack	end-to-end coding ability
Interactive agent benchmark	action trajectory	evaluated agent stack	agent plus scaffold
Process or critique benchmark	score, label, or critique	usually no fixed downstream executor	local step quality
Loop-system benchmark	loop or harness configuration	coding model inside each submitted system	complete loop system
LoopArena Type I	one candidate instruction	no Worker execution during evaluation	single control decision
LoopArena Type II	repeated Loop Contracts	fixed Worker on a selected task slice	runtime loop control on a selected task slice

Table 16: Comparison by evaluated object. The three LoopArena settings use different amounts of execution but all target the Controller’s loop-control ability.

Component	Prompt composition
Worker	The shared Worker system prompt, the task instruction, and either the bootstrap instruction or a Controller-derived continuation.
Reporter	The Reporter system prompt and a reporting request containing the task instruction and quoted Worker conversation.
Controller	The Controller system prompt followed by successive rendered Evidence Packets in one persistent Controller conversation.
Type I	The Controller decision policy with a selection-specific instruction, followed by one rendered Packet, four candidate decisions, and the answer instruction.
Reference policies	No control uses the shared Worker prompt with an autonomous-start instruction. Fixed control uses the shared Worker prompt and a deterministic continuation at each handoff; it invokes neither a Reporter nor a model Controller.

Table 17: Prompt composition in LoopArena. Benchmark instances provide the task, current state, reported evidence, and candidate decisions; the following sections give the fixed prompts and templates.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 463 papers

Jul 31, 2026

LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.

Han Li, Zhemin Fang, Rili Feng +8

Microsoft · Nanjing University · Shanghai Jiao Tong University +1

AI Coding Agents Software Engineering Benchmarks Long-Horizon Agent Evaluation

Ask PDF

Oct 10, 2026

TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

AI coding agents operate in iterative loops: they modify code, run tools, inspect the results, and decide what to do next. We call the design of this process loop engineering, which includes context management, model selection, compaction, and termination. Existing efficiency work mainly compresses tool observations or routes selected turns to cheaper models, but the practical opportunity for these optimizations remains unclear. We analyze 600 publicly released coding-agent trajectories spanning two agent scaffolds and two language models. Median observations contain only 118-286 tokens, while the large build logs commonly used to motivate compression occur near the ninety-ninth percentile. Moreover, 98.6% of input tokens are served from a prefix cache, so flat input pricing overestimates loop cost by roughly 7x on average under near-ideal caching. We derive and empirically validate a closed-form break-even condition for model delegation. Delegation increases cost in half of the evaluated settings, near the predicted boundary. More broadly, turn count explains 84% of the variance in log trajectory cost, compared with 4% for mean observation size; the most expensive trajectories have the most turns despite having the smallest observations. These associations suggest that observation-focused methods target a secondary cost factor, and identify unnecessary turns as the larger and largely unexamined opportunity, though we do not establish that turns can be removed while holding task success fixed. We release TurnCost, a tool for re-scoring trajectory corpora under alternative caching, pricing, compression, and routing policies.

Pramod Kumar, Raj Kumar Pandey

The Pennsylvania State University, University Park, USA

LLM Agent Workflow Optimization Coding Agents Cost-Aware Inference LLM Routing

Ask PDF

Oct 26, 2026

Benchmarking Coding Agents under Evolving Requirements

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Jia Feng, Alyssa Wu, Cuiyun Gao +3

Hunyuan Team, Tencent · The Chinese University of Hong Kong

Coding Agent Benchmarks Code Generation Process-Based Agent Evaluation LLM Agent Reliability

Ask PDF

Abstract

LLM Agent Orchestration

Explore similar work

LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

Benchmarking Coding Agents under Evolving Requirements