SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

Wei Minn; Zhensu Sun; Lwin Khin Shar; David Lo

cs.CLOct 8, 2026

SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

Authors: Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo

Organizations: School of Computing and Information Systems Singapore Management University

Abstract

Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are created at evaluation time, so no content exists in advance to leak, and its difficulty is set by the participating agents rather than fixed at release. We introduce SWE-Duel, an arena in which two coding agents evaluate each other over mirrored Red-Blue turns. In each turn, the Red agent works on a real repository across three context-isolated sessions. It proposes and implements a new feature with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests. The resulting pull request is admitted only if (1) it passes the feature tests and the repository's existing tests while failing exactly the bug tests, and (2) a fresh session of the same agent can repair the bug blind, which rules out unsolvable tasks. The Blue agent then reviews the pull request without being told a bug exists. It wins only if its revision passes all three test suites, and then the two agents swap roles. We run a tournament among 11 model-harness participants on 15 repositories in five languages for 1,500 turns. The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviews. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate. Finally, within one model family, tasks written by the newest version defeat a fixed-reference solver far more often than those of older versions (42.9% vs. 4.0%), giving preliminary evidence that task difficulty scales with task authoring capability.

Figures & tables

Figure 2: A contamination-resistant evaluation in SWE-Duel . Red agent implements a new feature with an embedded bug, so that the Blue agent cannot simply revert the repository to its original version to remove the bug as doing so would remove the implemented feature. Feature retention is deterministically verified with the feature test suite, while bug removal is deterministically verified with the bug test suite. Existing test suite verifies no regression by the Blue agent’s patch.

Figure 3: Three-session task generation. Each of the $n$ targets is stored inside a slot. Each slot permits at most five attempts: a challenge successfully generated within 5 attempts is stored inside its corresponding slot, whereas if it exceeds five attempts, it leaves an exhausted slot. Validation gate A ensures the verifiability of the implemented feature. Validation gate B ensures the verifiability of the embedded bug. Validation gate C ensures the fairness of the embedded bug by verifying the same-system reviewer detects and removes the bug from the new feature.

Competitor	Matches played	Won	Drawn	Lost	Red win (%)	Blue win (%)	Bradley–Terry
GPT-5.5 [codex]	10	8	1	1	11.3	98.7	+2.82
Claude Opus 4.8 [mini-swe-agent]	10	8	0	2	27.3	88.7	+2.46
GPT-5.5 [mini-swe-agent]	9	5	3	1	10.4	97.0	+2.13
Kimi-K2.7-Code [mini-swe-agent]	9	5	2	2	15.6	90.4	+1.82
DeepSeek-V4-Pro [mini-swe-agent]	9	5	1	3	18.5	91.9	+1.50
GLM-5.2 [mini-swe-agent]	9	4	2	3	14.8	89.6	+1.19

Table 1: Main tournament results. The rows are sorted in descending order by the full-schedule Bradley-Terry log-strength rating in the highlighted column. Full-schedule means the missing-Red auto-wins are retained. Figure 5 shows the trajectory of the rankings throughout round-robin tournament rounds before arriving at this final ranking.

Participant	$n$ tasks	Endog. Red wins	Endog. rate	Ref. Red wins/ $n$	Ref. rate (95% Wilson)
Claude Opus 4.8 [mini-swe-agent]	15	41/150	27.3%	4/15	26.7% $[10.9,\,52.0]$
DeepSeek-V4-Pro [mini-swe-agent]	15	25/135	18.5%	3/15	20.0% $[7.0,\,45.2]$
Qwen3.7-Max [mini-swe-agent]	15	18/135	13.3%	3/15	20.0% $[7.0,\,45.2]$
Nemotron-3-Ultra [mini-swe-agent]	7	10/70	14.3%	1/7	14.3% $[2.6,\,51.3]$
GLM-5.2 [mini-swe-agent]	13	20/117	17.1%	1/13	7.7% $[1.4,\,33.3]$
GPT-5.5 [Codex CLI]	13	17/130	13.1%	1/13	7.7% $[1.4,\,33.3]$

Table 3: Fixed reference task solver over the tasks used in the main tournament. The endogenous Red win rate is its Red success over its contested-only tournament turns; the reference Red win rate is the fraction of its admitted tasks that the fixed reference solver fails to solve blind.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Repository	Commit	Language	SLoC	Tests	Time (s)
Flask	7fff56f5172c	Python	13,751	480	3.7
Jinja	15206881c006	Python	18,903	909	4.2
SQLAlchemy	5237d923ddb0	Python	511,172	18,020	161.8
chi	3b171578ca44	Go	9,923	254	36.1
golang-jwt/jwt	7ceae619e739	Go	5,503	201	5.6
gorilla/csrf	9dd6af1f6d30	Go	2,760	50	3.9

Appendix

Table 4: The fixed repository field. Commit is the 12-hex prefix of the checked-out SHA. SLoC gives a rough estimate on the size/complexity of the repository via the number of nonblank lines in Git-tracked files of the repository’s implementation language, including tests but excluding documentation and build outputs. Tests is the number of test cases in existing-suite;

\dagger

denotes configured executable targets when the runner does not expose case totals. Time is the median wall-clock time of those existing-suite runs; commands that compile include compilation time.

Model	OpenRouter model ID	First access	Last access
GPT-5.5	openai/gpt-5.5	2026-06-18	2026-08-29
Claude Opus 4.8	anthropic/claude-opus-4.8	2026-06-18	2026-08-29
Kimi K2.7 Code	moonshotai/kimi-k2.7-code	2026-06-18	2026-08-29
DeepSeek V4 Pro	deepseek/deepseek-v4-pro	2026-06-18	2026-09-19
GLM 5.1	z-ai/glm-5.1	2026-09-18	2026-09-18
GLM 5.2	z-ai/glm-5.2	2026-06-18	2026-08-30

Appendix

Table 5: Per-model API access dates. Each entry gives the first and last recorded model call through the OpenRouter API (UTC). All calls of a model use the same OpenRouter model identifier regardless of the paired harness.

Figure 5: BT rank recomputed after each scheduled tournament round. A late entrant begins only when it first plays. This arrival-order diagnostic is not a ranking rule or convergence criterion.

Participant	$n$	3/3	2/3	1/3	0/3	Repairs/runs	Rate (95% Wilson)
DeepSeek-V4-Pro [mini-swe-agent]	4	4	–	–	–	12/12	100.0% $[75.7,\,100.0]$
Kimi-K2.7-Code [mini-swe-agent]	4	4	–	–	–	12/12	100.0% $[75.7,\,100.0]$
GPT-5.5 [Codex CLI]	4	4	–	–	–	12/12	100.0% $[75.7,\,100.0]$
GPT-5.5 [mini-swe-agent]	4	4	–	–	–	12/12	100.0% $[75.7,\,100.0]$
Claude Opus 4.8 [Claude Code]	4	3	1	–	–	11/12	91.7% $[64.6,\,98.5]$
Gemini-3.1-Pro [mini-swe-agent]	4	3	1	–	–	11/12	91.7% $[64.6,\,98.5]$

Appendix

Table 7: Session-C admission-noise stability study over the headline tournament bank. For each participant, its four sampled tournament tasks distribute over the repaired-

x/3

buckets according to how many of the

k{=}3

fresh blind re-runs of the exact Session-C probe repaired them; “Repairs/runs” pools the

44\times 3=132

per-run verdicts with Wilson 95% intervals.

Figure 6: Session-C admission-noise stability study. Each row stacks, as a share of its sampled tasks, how many of the three fresh blind re-runs repaired them (segment labels give absolute counts;

n

at each bar tip gives the row’s sample size); the right column reports the pooled per-run repair rate, and the top row aggregates all

44

tasks. The distribution concentrates on reliable solvability (

3/3

), with a single lucky admission (

0/3

Figure 7: Task difficulty: scheduled field vs. fixed reference. Each point is one participant’s Red win rate against the full scheduled solver field (x-axis) and against the fixed reference solver (y-axis). Error bars are Wilson 95% intervals. Points without a harness tag use mini-swe-agent.

Full rank	Competitor	Solving BT	95% CI	Solving success	Admitted / scheduled slots	Coverage
1	GPT-5.5 [codex]	+2.19	$[+1.35,\,+4.89]$	98.3%	13 / 15	86.7%
2	Claude Opus 4.8 [mini-swe-agent]	-0.32	$[-0.81,\,+0.08]$	85.7%	15 / 15	100.0%
3	GPT-5.5 [mini-swe-agent]	+1.45	$[+0.82,\,+2.82]$	96.4%	13 / 15	86.7%
4	Kimi-K2.7-Code [mini-swe-agent]	+0.09	$[-0.42,\,+0.59]$	88.3%	13 / 15	86.7%
5	DeepSeek-V4-Pro [mini-swe-agent]	+0.25	$[-0.28,\,+0.79]$	89.9%	15 / 15	100.0%
6	GLM-5.2 [mini-swe-agent]	+0.01	$[-0.42,\,+0.47]$	87.4%	13 / 15	86.7%

Appendix

Table 8: Task solving and generation-coverage decomposition for the headline tournament. Participants are shown in the order of full-schedule tournament’s BT ratings. Solving BT is a role-conditioned fit on contested Blue turns only. Coverage is admitted slots divided by the 15 repository slots.

Figure 8: Task-clustered contested task-creation and task-solving success by competitor. Missing-Red auto-wins are excluded.

Figure 9: Red–Blue transfer matrix, ordered by the tournament’s Bradley–Terry log-strength rating. Each observed cell reports the Red win rate as a percentage and as raw wins over the 15 scheduled repository turns ( $n=15$ ).

Competitor	Full-schedule turn BT (95% CI)	Full-schedule 95% rank interval	Contested-only turn BT (95% CI)	Contested-only 95% rank interval
Claude Opus 4.8 [mini-swe-agent]	$+0.30$ $[-0.10,\,+0.80]$	1–8	$+0.12$ $[-0.34,\,+0.61]$	1–10
DeepSeek-V4-Pro [mini-swe-agent]	$+0.25$ $[-0.15,\,+0.73]$	1–8	$+0.07$ $[-0.31,\,+0.54]$	1–10
GPT-5.5 [mini-swe-agent]	$+0.19$ $[-0.20,\,+0.72]$	1–9	$+0.17$ $[-0.28,\,+0.66]$	1–9
GPT-5.5 [codex]	$+0.18$ $[-0.20,\,+0.73]$	1–9	$+0.15$ $[-0.29,\,+0.84]$	1–9
Kimi-K2.7-Code [mini-swe-agent]	$+0.17$ $[-0.23,\,+0.68]$	1–9	$+0.13$ $[-0.28,\,+0.62]$	1–10
Qwen3.7-Max [mini-swe-agent]	$+0.14$ $[-0.28,\,+0.65]$	1–9	$-0.05$ $[-0.46,\,+0.46]$	1–10

Appendix

Table 9: Turn-level Bradley–Terry ratings for the main tournament. BT values are point estimates with 95% percentile intervals from 1,000 task-cluster bootstrap draws; rank intervals are central 95% intervals over the same draws. The full-schedule variant treats missing-Red auto-wins as Blue wins; the contested-only variant drops them.

Harness	W	D	L	Score	Rate (95% CI)	Red win	Blue win	BT (95% CI)
mini-swe-agent	4	1	1	93/180	$0.52\,[0.43,\,0.60]$	0.13	0.90	$+0.73\,[-0.25,\,+5.56]$
Codex CLI	2	1	3	90/180	$0.50\,[0.42,\,0.60]$	0.11	0.89	$-0.25\,[-3.50,\,+1.34]$
Claude Code	1	2	3	87/180	$0.48\,[0.40,\,0.58]$	0.09	0.88	$-0.49\,[-3.83,\,+0.65]$

Appendix

Table 10: Harness ablation aggregated over three fixed base models. Score sums both roles over the full schedule; Rate is the score rate with a 95% task-clustered percentile interval; Red win and Blue win are per-role turn success rates over the same full schedule. BT is the match-level Bradley–Terry log-strength fit to the nine same-model sub-match outcomes pooled across the base models, with a 95% percentile interval from the ordinary match bootstrap used by the headline rating.

Figure 12: Harness ablation controlled by base model. Composite two-role score rate for each fixed base model under each of the three harnesses; whiskers are 95% task-clustered percentile intervals (60 turns, 45 clusters per bar).

Harness A	Harness B	A score	B score	Margin (95% CI)	Models
Claude Code	Codex CLI	46	44	$+2\,[-16,\,+22]$	3
Codex CLI	mini-swe-agent	46	44	$+2\,[-16,\,+20]$	3
Claude Code	mini-swe-agent	41	49	$-8\,[-28,\,+12]$	3

Appendix

Table 11: Aggregate score for each harness pair. Margin is (A

-

B) points with a 95% clustered percentile interval on the paired per-turn difference. The direct comparisons resolve no ordering at this sample size and the apparent cycle in pair totals is consistent with sampling noise.

Competitor	Gen $	Val $	Eval $	Total $
GPT-5.5 [codex]	149.56	47.54	444.79	641.89
GPT-5.5 [mini-swe-agent]	48.38	17.15	181.65	247.18
Claude Opus 4.8 [mini-swe-agent]	44.93	22.12	156.54	223.59
Gemini-3.1-Pro [mini-swe-agent]	106.18	19.77	61.81	187.76
Claude Opus 4.8 [claude-code]	42.31	14.04	122.70	179.05
Nemotron-3-Ultra [mini-swe-agent]	84.66	2.85	83.82	171.34

Appendix

Table 12: Generation, blind self-review validation, and Blue evaluation API cost by competitor. Generation cost includes failed attempts in slots that never produced a task, preventing successful-only accounting.

Figure 13: Secondary efficiency diagnostic relating the tournament’s Bradley–Terry log-strength rating to API cost and output-token use. These observational associations combine model pricing, generation coverage, and behavior; they are not a controlled efficiency comparison.

Competitor	Turns	Success	Output M	Median k	$r$ [95% CI]
Claude Opus 4.8 [mini-swe-agent]	119	0.857	0.75	5.9	$-0.04$ $[-0.21,+0.15]$
Claude Opus 4.8 [claude-code]	118	0.839	0.80	6.9	$-0.18$ $[-0.35,+0.00]$
GPT-5.5 [mini-swe-agent]	111	0.964	0.84	7.3	$-0.11$ $[-0.20,-0.00]$
GPT-5.5 [codex]	120	0.983	0.81	6.1	$+0.09$ $[+0.00,+0.18]$
DeepSeek-V4-Pro [mini-swe-agent]	109	0.899	1.15	10.1	$-0.01$ $[-0.23,+0.21]$
GLM-5.2 [mini-swe-agent]	111	0.874	0.78	6.6	$+0.04$ $[-0.14,+0.22]$

Appendix

Table 13: Contested Blue task-solving resource use. “Turns” is the number of all contested Blue turns for a given participant. “Success” is the rate of successful task solving out of all turns; “Output M” is the total output tokens over all contested turns, in millions; “Median k” is the median per-turn output tokens, in thousands. The final column is Pearson’s correlation between binary task-solving success and

\log(1+\text{output tokens})

, with a task-cluster bootstrap 95% interval.

Rank	Competitor	Solving BT	95% interval	Rank interval	Median turns
1	GPT-5.5 [codex]	+2.60	$[+2.40,+4.94]$	1–1	94
2	GPT-5.5 [mini-swe-agent]	+1.38	$[+0.97,+2.41]$	2–2	84
3	DeepSeek-V4-Pro [mini-swe-agent]	+0.17	$[-0.32,+0.86]$	3–7	61
4	Qwen3.7-Max [mini-swe-agent]	+0.15	$[-0.21,+0.57]$	3–6	83
5	Kimi-K2.7-Code [mini-swe-agent]	+0.02	$[-0.44,+0.56]$	3–7	68
6	GLM-5.2 [mini-swe-agent]	-0.08	$[-0.42,+0.28]$	3–7	91

Appendix

Table 14: Output-token-budget-matched contested task-solving ranking at 638,976 tokens per participant. This task-solving-only estimand excludes Red task creation, uses provider-reported output tokens, and is not meant to replace the main tournament ranking.

Main tournament
Cost component	USD	Share
Task creation (incl. failed attempts)	669.69	31.8%
Blind self-review validation	158.34	7.5%
Blue evaluation	1281.18	60.7%
Tournament total	2109.21	100.0%
Entrance: Gemini-3.1-Pro [mini-swe-agent]

Appendix

Table 15: API-cost ledger: the main tournament’s cost split and the marginal cost of the late entrance. Shares are fractions of the tournament total; per-participant totals are in Table 12 .

Figure 14: Textual interface for task generation. Rows correspond to numbered generation slots; each row tracks the feature/bug sessions, gate outcomes, and attempt history of that slot.

Figure 15: Textual interface for tournament scheduling. Pairings are drawn from the persisted schedule, solving runs are served from the cache, and completed matches report per-side outcomes and cost.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 389 papers

Jul 8, 2026

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Wenqi Huang, Charley Lee, Leonard Tng +1

1Datacurve

AI Coding Agents Software Engineering Benchmarks Long-Horizon Agent Evaluation Coding Agent Benchmarks

Ask PDF

Oct 16, 2026

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.

Soham Dan, Jeff Da, Sanyam Bhutani +8

Scale AI · Reflection AI

Coding Agent Benchmarks Software Engineering Benchmarks Code Generation Benchmarks Long-Horizon Agent Evaluation Software Engineering Agents

Ask PDF

Oct 21, 2026

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These results indicate that single-turn performance is insufficient to characterize interactive coding ability.

Zhan-Yu Wu, Ji Luo, Alyssa Wu +5

Tencent Hunyuan

Coding Agents AI Agent Evaluation Coding Agent Benchmarks Software Engineering Benchmarks Repository-Level Program Repair

Ask PDF

Abstract

Benchmark Design

Explore similar work

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users