UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

Zhan-Yu Wu; Ji Luo; Alyssa Wu; Junjielong Xu; Richeng Xuan; Zhichao Hu; Chongming Gao; Xinting Hu

cs.AIOct 21, 2026

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

Authors: Zhan-Yu Wu, Ji Luo, Alyssa Wu, Junjielong Xu, Richeng Xuan, Zhichao Hu, Chongming Gao, Xinting Hu

Organizations: Tencent Hunyuan

Abstract

As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These results indicate that single-turn performance is insufficient to characterize interactive coding ability.

Figures & tables

Figure 1: Existing multi-turn benchmarks keep the simulated user in one persona, while in the real development sessions we measure a developer’s style shifts from turn to turn. On the same date-off-by-one bug, the static simulator phrases every turn as precise guidance, while the UserSWE simulator opens on a vague complaint, guesses a wrong cause, and reaches guidance only at the third turn.

Benchmark	Recent tasks	Multi-turn	Persona	Judge method	Judge freedom
SWE-bench	$\times$	$\times$	$\times$	tests	Low
SWE-bench Pro	$\times$	$\times$	$\times$	tests, hand-checked	Low
SWE-bench-Live	✓	$\times$	$\times$	tests, auto-validated	Low
SWE-Evo	$\times$	$\times$	$\times$	tests	Low
SWE-ContextBench	$\times$	$\times$	$\times$	tests	Low
SWE-Interact	$\times$	✓	static	source verifier	Low

Table 1: A comparison of recent benchmarks ( Jimenez et al., 2024 ; Deng et al., 2026 ; Zhang et al., 2025 ; Le et al., 2025 ; Zhu et al., 2026 ; Raghavendra et al., 2026 ; Wu et al., 2026 ) with UserSWE.

Figure 2: The UserSWE pipeline. The problem panel shows how a task is constructed, the persona panel how the simulated user is built, and the evaluation panel how a session is run and scored.

	no cause	correct	off-target	wrong
clear	Exported dates are one day early.	Exported dates are one day early. // Timezone converted twice?	Exported dates are one day early. // Maybe the template?	Exported dates are one day early. // Timezone is fine, it is the template.
vague	The export looks off.	The export looks off. // Timezone converted twice?	The export looks off. // Maybe the template?	The export looks off. // Timezone is fine, it is the template.

Table 2: How one symptom, dates coming out of the export a day early because the timezone conversion runs twice, is put under the eight personas. Columns give the four ways of stating a cause, rows the two ways of describing the symptom.

	Resolved within $k$ turns
Model	1	2	3	4	5	6	7	8	$k$	LDPR
Claude Opus 5	25.0	56.9	66.7	76.4	79.2	80.6	80.6	80.6	2.99	0.640
Claude Fable 5.1 ∗	31.9	47.2	62.5	66.7	75.0	75.0	75.0	76.4	3.14	0.593
Grok-4.6	18.1	43.1	58.3	66.7	76.4	79.2	79.2	80.6	3.40	0.569
GPT-6-Astra	12.5	38.9	55.6	70.8	77.8	79.2	80.6	81.9	3.44	0.561
DeepSeek-v4.1-Flash	18.1	40.3	51.4	65.3	73.6	76.4	76.4	76.4	3.47	0.537

Table 3: Evaluation results on UserSWE. The middle columns give the resolved rate within that many turns.

Figure 3: The order of the four leading agents on published single-turn tasks, and how that order changes as the turn budget grows on UserSWE.

Model	Clear	Dynamic	$\Delta$	Clear	Dynamic	$\Delta$
	LDPR			$k$
Claude Opus 5	0.688	0.668	$\downarrow$ .020	2.27	2.50	$\uparrow$ .23
GPT-6-Astra	0.656	0.580	$\downarrow$ .076	2.54	2.92	$\uparrow$ .38
DeepSeek-v4.1-Flash	0.628	0.562	$\downarrow$ .066	2.58	2.88	$\uparrow$ .30
Kimi-K3	0.428	0.470	$\uparrow$ .042	3.46	3.25	$\downarrow$ .21

Table 4: Persona ablation under a clear-and-correct user and the dynamic persona chain, over the same 50 single-issue tasks.

\Delta

is the change from clear-and-correct to dynamic.

Reviewer	Rewrite	Evidence	Guidance	Equivalence	Persona
Codex / GPT-5.6-Sol	62.4%	70.3%	14.3%	9.5%	5.9%
Claude Code / Claude Opus 5	25.4%	42.1%	41.5%	5.3%	11.1%

Table 5: Reviewer rewrites and their objections. Rewrite is the share of reviewed turns rewritten within each configuration; the four objection columns break down that configuration’s own objections and sum to 100% across each row.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

State	Test set	Required	Establishes
base	pass-to-pass	pass	existing behavior is intact
base	fail-to-pass	fail	the task remains open
gold	pass-to-pass	pass	the fix introduces no covered regression
gold	fail-to-pass	pass	the task is closeable

Appendix

Table 6: The four admission outcomes.

Cause	Opening behavior	Response to disagreement
None	Gives no diagnosis or repair direction.	Continues with observable symptoms and answers direct questions.
Correct	Offers the correct direction as a low-confidence guess.	Gives way when the agent provides a better account.
Off-target	Offers a plausible but mistaken direction.	Drops the direction after reasoned disagreement and continues as none .
Wrong	Opens with the same mistaken direction as off-target .	Holds it against reasoning and yields only to test, code or reproduction evidence.

Appendix

Table 7: The four settings on the cause axis. Each is paired with either a clear or vague symptom.

Symptom	Stated cause	Sessions	Share
clear	none	7,168	52.98%
clear	correct	2,575	19.03%
clear	off	122	0.90%
clear	wrong	197	1.46%
vague	none	3,103	22.93%
vague	correct	123	0.91%

Appendix

Table 8: Opening persona distribution over the 13,530 code-changing sessions. The eight cells cover 98.65% of them; the rest opened as other .

Figure 5: The eight turn-to-turn transition matrices, one for each opening persona. Rows are current personas and columns are next personas. Rows with fewer than 30 observed transitions are greyed and marked with an asterisk, and entries below 0.5% are left blank.

Priority	Trigger	Release
1	The agent asks a direct question.	Answer that question.
2	A user-observable core failure remains.	Release the most blocking observable symptom.
3	A core business boundary remains uncovered.	Release the relevant boundary.
4	Only a secondary edge case remains.	Release the relevant edge case.

Appendix

Table 9: Priority for selecting guidance from the information pool.

State	Name	Meaning	Framework action
U	unchanged	The agent asks or discusses without changing the workspace.	Return the user’s answer and continue; add 0.5 to the weighted turn count.
F	continue	At least one acceptance item remains unsatisfied.	Send the selected guidance and begin the next turn.
C	framework step	Automated items pass, but another framework step remains.	Run that step and return its evidence to the Judge.
D	done	All completion conditions are satisfied.	End the session and record the task as resolved.
R	equivalence	A behavior-equivalent test rewrite is requested.	Execute the rewrite through the equivalence gates, then settle again.
E	judge error	The Judge fails irrecoverably.	Stop and mark the run failed.

Appendix

Table 10: Judge states.

Context	Action	Framework effect
Turn review	KEEP	Dispatch the Judge candidate unchanged.
Turn review	REWRITE	Replace it with a complete response carrying the revised state, verdicts, evidence and user message.
Turn review	UNAVAILABLE	Stop the run rather than dispatch an unreviewed candidate.
Equivalence	APPROVE	Allow a mechanically valid rewrite to update the requested items.
Equivalence	REJECT	Keep the canonical failure even when the mechanical gates pass.

Appendix

Table 11: Reviewer actions and their framework effects.

Item	Scope
leak_test	Hidden tests, acceptance lists or test-side secrets reached the agent.
leak_issue	The prompt exposed wording or links from the source issue or pull request.
issue_allowed	The word issue appeared only in its ordinary product or repository meaning and was harmless.
judge_test_blocked	The Judge reported that the canonical test command could not be executed.
test_bridge_broken	The test bridge or environment produced no trustworthy test signal.
manual_pytest_workaround	The Judge bypassed the canonical runner with an ad hoc test command.

Appendix

Table 12: Post-run audit rubric. A flag records a condition for review and does not by itself make the score untrustworthy.

Model	Scaffold
Claude Opus 5	Claude Code
Claude Fable 5.1	Claude Code
Grok-4.6	Claude Code
GPT-6-Astra	Codex
DeepSeek-v4.1-Flash	DSH
GPT-5.6-Sol	Codex

Appendix

Table 13: The model and scaffold behind each agent.

Model	LDPR	95% interval
Claude Opus 5	0.640	[0.553, 0.722]
Claude Fable 5.1	0.593	[0.499, 0.684]
Grok-4.6	0.569	[0.486, 0.649]
GPT-6-Astra	0.561	[0.481, 0.639]
DeepSeek-v4.1-Flash	0.537	[0.450, 0.621]
GPT-5.6-Sol	0.500	[0.412, 0.587]

Appendix

Table 14: LDPR with a 95% interval from resampling tasks with replacement.

	Agent			Judging side				Edit size
Model	Requests	Tokens	Cost	Requests	Judge	Reviewer	Wall	Files	Lines
Claude Opus 5	178	22279	170.4	71	33.5	30.7	128	5	572
Claude Fable 5.1	273	38635	710.5	75	46.4	34.8	675	6	633
Grok-4.6	177	18175	110.8	82	40.4	37.1	168	6	696
GPT-6-Astra	158	13725	264.2	114	61.9	38.8	805	7	723
DeepSeek-v4.1-Flash	386	28355	4.3	134	72.3	30.4	278	3	184

Appendix

Table 15: Per-task cost, separated between the agent and judging. Tokens are in thousands and exclude cache reads; costs are in CNY. Metrics use available telemetry: agent columns are means, while judging costs and edit sizes are medians.

Model	Reviewer	Rewrite	Evidence	Guidance	Equiv.	Persona
Claude Opus 5	Claude Code	30.0%	32.6%	44.0%	9.3%	14.1%
	Codex	64.8%	70.9%	13.9%	7.1%	8.1%
Claude Fable 5.1	Claude Code	24.2%	48.1%	30.1%	7.9%	13.9%
	Codex	70.5%	66.3%	17.4%	11.7%	4.6%
Grok-4.6	Claude Code	21.9%	40.8%	38.5%	12.9%	7.8%
	Codex	65.3%	78.8%	12.7%	6.4%	2.1%

Appendix

Table 16: Reviewer rewrites by leaderboard model and Reviewer. Rewrite is the share of reviewed turns rewritten; the remaining columns break down that row’s own objections and sum to 100% across each row.

First failed gate	Condition	Requests
Candidate execution	Rewritten acceptance tests fail, or a pass-to-pass test regresses, on the candidate workspace.	47
Base discrimination	The rewritten test also passes on the unmodified base workspace.	6
Behavioral review	Both execution gates pass, but the rewrite drops behavior required by the issue.	9

Appendix

Table 17: First failed gate among refused equivalence requests.

	Phase 1 (#2376)	Phase 2 (#2377)
User turn	Generating a launcher for a Docker-runtime package produces npx instead, and it reports the package cannot be found. I suspect the runtime type is not recognized and the package falls through to the generic launcher branch.	For the same Docker package, generating a configuration with --target vscode makes my custom Docker options disappear. I suspect the per-target conversion loses the Docker arguments.
Acceptance	Registry name maps to Docker, and the target generates a Docker launcher carrying the image.	Value entries become docker run options, and the launcher is no longer the bare fallback.
Revealed	At the opening.	Only after both first-phase points pass.

Appendix

Table 18: The two phases of apm-casc-2376-2377 . The user turns are the frozen utterances for a clear-symptom, correct-cause persona, translated from the deployment language. Both were approved at the breakpoint review.

Turn	User turn	Agent response	Tests after the turn
1	The generated date object cannot be dumped to JSON, and an option returning a string would help.	Adds as_string , which returns a fixed ISO string.	The regression guard stays green and the feature acceptance points stay red.
2	The output format has to be selectable.	Lets as_string take a format string.	Unchanged, with the feature acceptance points still red.
3	Calls that pass pattern are still rejected.	Adds pattern to the affected date APIs and propagates it through the call chain, keeping the object return when it is omitted.	Every feature acceptance point turns green and the regression guard stays green.

Appendix

Table 19: The three turns of add-date-provider-pattern-arg . Tests run after every turn, and what the simulated user says next is selected from the information pool against that result.

Request	Rewrite	Ruling
Granted	Replace a literal comparison against the task dictionary with a non-default command-line timeout, load the resulting play, and inspect the effective timeout on its task object.	Granted. The base state still failed, the candidate and regression tests passed, and the Reviewer found the command-line-to-task behavior equivalent. The Reviewer separately noted dropped assertions on asynchronous fields and an unrelated wording edit.
Refused	Drive the command through sys.argv and capture values received by a wrapped function instead of checking the public parse_args(args) and run() return contracts.	Refused. The mechanical gates passed, but the rewrite removed behaviors named by the contract, introduced mocks the contract excluded, and replaced an optional list of paths with a scalar path.

Appendix

Table 20: Two equivalence requests and the behavior retained by their rewritten tests.

Domain	What the Judge wrote	How the Reviewer settled it
Evidence	A zero return code together with a collection-time import error, neither of which the canonical test window shows.	Recorded what that window holds, a non-zero return code with the fail-to-pass set failing and the pass-to-pass set passing, and anchored the diagnosis to the workspace changes.
Guidance	A next message carrying evaluator detail, which the Judge is required to keep separate from its evidence, and a candidate state the framework could not read.	Restored the state to where the framework reads it and cut the next message to one user-visible observation, that the saved name still carried Mr.

Appendix

Table 21: The two objections filed on turn one of strip-author-name-honorifics and how the Reviewer settled each.

Model	Clear	Vague	$\Delta$
Claude Opus 5	0.671	0.540	+.131 $\pm$ .106
Claude Fable 5.1	0.656	0.388	+.268 $\pm$ .123
Grok-4.6	0.599	0.474	+.125 $\pm$ .101
GPT-6-Astra	0.603	0.423	+.181 $\pm$ .102
DeepSeek-v4.1-Flash	0.516	0.604	$-$ .088 $\pm$ .109
GPT-5.6-Sol	0.549	0.341	+.208 $\pm$ .101

Appendix

Table 22: LDPR by the clarity of the opening symptom, over the 55 tasks whose opening states the symptom precisely and the 17 that state it as an impression.

\Delta

is the difference between the two groups, with its standard error.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 140 papers

Jun 29, 2026

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.

Yifan Wu, Zhuokai Zhao, Songlin Li +8

Coding Agents AI Agent Evaluation AI Agent Benchmarks Human-AI Collaboration

Ask PDF

Oct 21, 2026

SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill. To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment. To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent. We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task. The entire framework is embedded in a Harbor environment, which orchestrates the coding agent, user simulator, and conversation validator, and executes the test environment to determine whether the agent successfully solves the task. Our results and source code are available in our GitHub repository (https://github.com/lorenzbrehme/swe-dive_usersim).

Lorenz Brehme, Benjamin Steenhoek, Ruth Breu

University of Innsbruck Innsbruck, Austria · Microsoft Redmond, WA, USA

Coding Agents AI Agent Evaluation AI Agent Benchmarks User Simulation

Ask PDF

Jun 29, 2026

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.

Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal +1

1Scale AI

Coding Agents Software Engineering Benchmarks AI Agent Evaluation User Simulation

Ask PDF

Abstract

Coding Agents

Explore similar work

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions