Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at https://github.com/upwork/SWE-Clar.
Figures & tables
Figure 1: A fictional commission illustrates the motivation for SWE-Clar in greenfield software development in freelance settings: (1) long development makes continuous client feedback difficult to sustain; (2) complex requirements can be misunderstood even after receiving replies.
Figure 2: SWE-Clar separates client interaction from two independent evaluations. The coding agent interacts with a client agent whose replies are reviewed by a supervisor agent. The resulting clarification context initializes both branches. Acceptance tests determine the implementation score; answers to requirement questions determine the clarification score. Assessment questions, answers, and scores do not enter the implementation branch.
Figure 3
Benchmark
Task setting
Client interaction
Evaluation
SWE-Lancer ( 2025 )
Upwork commissions; existing-codebase changes
No client clarification
End-to-end code tests
ClarifyCodeBench ( 2026 )
LiveCodeBench; programming problems
Questions before code generation
Question quality and efficiency; code tests
Ambig-SWE ( 2026 )
SWE-bench Verified; issue resolution
Queries to a simulated issue reporter
Information recovery; patch tests
SWE-Together ( 2026 )
Recorded coding sessions; repository tasks
Feedback throughout development
Intent coverage, user correction, and repository correctness
SWE-Interact ( 2026 )
Existing SWE benchmarks; repository tasks
Progressive disclosure and implementation feedback
Plan/code goal coverage; task verifiers
ICAE-Bench ( 2026 )
Open-source repositories; from-scratch generation
Requirement queries during project building
Constraint coverage; black-box code tests
Table 1: Representative coding benchmarks compared by task setting, client interaction, and evaluation. Independent requirement QA scores the coding agent’s answers in a context isolated from code implementation. SWE-Lancer refers to its individual-contributor coding tasks.
Coding model
Implementation score (%)
Clarification score (%)
GPT-6 Astra
67.0 [58, 76]
87.5 [85.0, 90.1]
Claude Opus 5
63.0 [53, 72]
84.0 [80.9, 86.8]
Claude Fable 5.1
60.0 [50, 69]
84.3 [81.0, 87.3]
GPT-5.6 Sol
57.0 [47, 67]
78.1 [74.9, 81.3]
Qwen-3.8 Max
55.0 [45, 64]
81.3 [77.9, 84.7]
GLM-5.3
54.0 [44, 64]
76.8 [73.3, 80.2]
Table 2: SWE-Clar evaluation on 100 tasks per model. Bold and underlined values indicate the highest and second-highest scores; higher is better. Brackets show 95% task-bootstrap confidence intervals (10,000 resamples).
Figure 5: Effect of requirement availability on the two scores. Each model is compared on 100 tasks under each condition. The same grading rubric is used across conditions and in the main comparison.
Figure 6: Code-test pass rates within answer categories. Each bar uses the RUs in that model’s answer category as its denominator. Incorrect includes contradictory or substantively unanswered requirements.
Inspect materials, ask questions, decide when to stop
Client
Complete requirements, task materials, approved conversation, current question
Draft or revise a reply within the requested scope
Supervisor
Same information as the Client, plus the draft reply
Approve the reply or request revision
Code implementation
Coding
Frozen context and development workspace
Implement, run development checks, submit software for acceptance testing
Clarification assessment
Coding
Frozen context, requirement questions, read-only development materials
Answer questions without further interaction or implementation
Judge agent
Reference requirements and answers, candidate answers, relevant acceptance tests
Grade requirement answers independently of code outcomes
Appendix
Table 3: Agent responsibilities and information access. Both evaluation branches start from the frozen clarification context. Private requirement materials are accessible only to client simulation and evaluation agents.
Label
Criterion
Score
Correct
States all material requirements correctly.
1
Partial
Establishes substantive correct behavior, with a material omission or uncertainty and no explicit material contradiction.
0.5
Incorrect
States incompatible behavior or provides no substantive correct answer.
0
Appendix
Table 4: Requirement-answer grading rubric.
Model
RU pass (%)
USD/task
GPT-6 Astra
88.5
5.391
Claude Fable 5.1
86.7
11.033
Claude Opus 5
87.2
9.694
GPT-5.6 Sol
84.4
1.965
Qwen-3.8 Max
85.9
1.703
GLM-5.3
82.4
1.393
Appendix
Table 5: Supplementary RU pass rates and mean costs for 100 evaluation tasks per model.
Figure 7: Cost and implementation score. Cost includes coding-agent calls during clarification and implementation.
Candidate
Information condition
Implementation score
Clarification score
RU pass rate
GPT-5.6 Luna
No client interaction
6.0
41.4
37.5
GPT-5.6 Luna
Standard protocol
49.0
71.8
82.7
GPT-5.6 Luna
Complete requirements
75.0
93.1
93.2
GLM-5.3 Flash
No client interaction
5.0
35.1
36.0
GLM-5.3 Flash
Standard protocol
39.0
73.2
77.8
GLM-5.3 Flash
Complete requirements
71.0
96.2
91.1
Appendix
Table 6: Information conditions on matched tasks within each candidate. Scores are percentages.
Candidate
Comparison
Difference
95% interval
GPT-5.6 Luna
Standard minus no interaction
43.0
[33.0, 53.0]
GPT-5.6 Luna
Complete minus standard
26.0
[14.0, 37.0]
GLM-5.3 Flash
Standard minus no interaction
34.0
[25.0, 43.0]
GLM-5.3 Flash
Complete minus standard
32.0
[20.0, 43.0]
Appendix
Table 7: Paired task-pass-rate differences in percentage points, with task-bootstrap 95% intervals.
Figure 8: Mean requirement disclosure after filtering conversations with outlying round counts. Models are ordered by their final proportion of fully disclosed hidden RUs. Additional horizontal-axis ticks mark each curve’s final round.
Model
Fully disclosed RUs
Partial (%)
Incorrect (%)
GPT-6 Astra
249
2.4
3.2
Claude Fable 5.1
247
2.8
4.9
Claude Opus 5
261
3.8
4.6
GPT-5.6 Sol
212
10.4
2.8
Qwen-3.8 Max
220
3.2
6.4
GLM-5.3
202
4.0
6.4
Appendix
Table 8: Partial and Incorrect answers among fully disclosed hidden RUs. Counts give the eligible RUs per model; remaining answers are Correct.
Model
Strict (%)
Correct
Partial
Incorrect
GPT-6 Astra
79.0
404
89
19
Claude Fable 5.1
76.1
388
85
39
Claude Opus 5
76.3
389
79
44
GPT-5.6 Sol
62.2
317
166
29
Qwen-3.8 Max
74.0
379
76
57
GLM-5.3
64.7
329
127
56
Appendix
Table 9: Strict answer correctness and answer-label counts over 100 tasks per model. Each model has 512 requirement answers.
Figure 9: Joint distribution of requirement answers and code-test outcomes. Cells show percentages of all 512 RUs per model; the six cells in each panel sum to 100%, up to rounding. All panels share one color scale and follow the main-table model order. Tests pass and Tests fail refer to acceptance tests for the corresponding RU.
Model
Clarification before coding
Continued clarification
Implementation score
Clarification score
Implementation score
Clarification score
GPT-5.6 Luna
49.0
71.8
26.0
59.0
GLM-5.3 Flash
39.0
73.2
13.0
50.9
Appendix
Table 10: Implementation and clarification scores (%) for two workflows on 100 tasks per model. Clarification scores measure requirement-answer performance.
Transition
Percentage (%)
Incorrect → correct
46.0
Correct → incorrect
2.9
Correct → correct
43.7
Incorrect → incorrect
7.5
Appendix
Table 11: First-draft to final-reply outcomes among the 561 revised questions. Values are percentages and sum to 100%, up to rounding.
Singapore Management University, Singapore · Monash University Malaysia, Malaysia · Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea +5