As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.
Figures & tables
Figure 1. One complaint describes incorrect assumptions leading to an undesirable implementation; the other describes time and tokens wasted as the model continues along a misunderstood path.
Benchmark
Time
Granularity
#Tasks
R. Character.
R. Clarification
HumanEval ( Chen et al., 2021 )
2021
Function-level
164
Clear&Complete
✗
MBPP ( Austin et al., 2021 )
2021
Function-level
974
Clear&Complete
✗
SWE-bench ( Jimenez et al., 2024 )
2023
Pull Request-level
2,294
Clear&Complete
✗
LiveCodeBench ( Jain et al., 2025 )
2024
Function-level
400+
Clear&Complete
✗
WebGen-Bench ( Lu et al., 2026 )
2025
Repository-level
101
Underspecified
✗
FeatureBench ( Zhou et al., 2026b )
2026
Repository-level
200
Clear&Complete
✗
Table 1. Comparison of ReqClarifyBench with representative code generation benchmarks in terms of publication year, task granularity, dataset size, requirement characteristics, and support for interactive requirement clarification. “R.” denotes “Requirement” and Character. denotes “characteristic”.
Figure 2. The overview of ReqClarifyBench construction pipeline.
Figure 3. Behavioral evaluation protocols of the three selected source benchmarks.
Source Benchmark
# Tasks
Avg. # Atomic Req.
Avg. Doc Tokens
Avg. Intent Tokens
MiniAppBench ( zhang et al., 2026 )
100
157
1,781
10
GameCraft-Bench ( Luo et al., 2026b )
140
240
3,526
12
WebGen-Bench ( Lu et al., 2026 )
101
165
1,383
17
ReqClarifyBench
341
193
2,380
13
Table 2. Statistics of ReqClarifyBench across its three source benchmarks, including the number of tasks, the average number of atomic requirements per task, and the average token lengths of detailed requirement documents and high-level user intents. “#” denotes the number of items.
Figure 4. Interactive requirement clarification and evaluation in ReqClarifyBench .
Type
Model
Publisher
Size
Time
Window Length
Proprietary
GPT-5.6-Sol ( OpenAI, 2026 )
OpenAI
–
2026-07
1.05M
Claude-Opus-5 ( Anthropic, 2026a )
Anthropic
–
2026-07
1M
Gemini-3.1-Pro ( Google DeepMind, 2026 )
Google
–
2026-02
1M
Open Weight
GLM-5.3 ( GLM-5-Team et al., 2026 )
Z.ai
753B
2026-08
1M
DeepSeek-V4-Pro ( DeepSeek-AI et al., 2026 )
DeepSeek
1.6T
2026-04
1M
Qwen3-235B-A22B ( Yang et al., 2025a )
Alibaba
235B
2025-04
128K
Table 3. Studied Large Language Models
Figure 5. Distribution of stopping rounds across the six evaluated LLMs under three user personas. Boxes show the interquartile range, center lines denote the median, and diamonds indicate the mean stopping round.
Figure 6. Clarification dynamics across interaction rounds for different models and user personas. For each model–persona pair, EQR and AvgClarified are reported only up to the third quartile (Q3) of the stopping-round distribution to avoid sparsely populated tail rounds.
Figure 7. Effect of clarification on BERTScore F1 and ReqAlign across different models.
Figure 8. Comparison of requirement-document quality and clarification effectiveness between base models and their coding-agent harness counterparts. EN. , VR. , and MC. denote Expert Nitpicker, Vague Requester, and Mind Changer, respectively. w/o clari. denotes directly generating the requirement document without interactive requirement clarification.
Benchmark
Score
Intent
Direct Doc.
Clarified Doc.
GameCraft-Bench
0.012
0.273
0.418
MiniAppBench
0.451
0.543
0.943
WebGen-Bench
0.271
0.384
0.659
Table 5. Code-generation performance across three source benchmarks under three input settings: Intent (the initial user intent), Direct Doc. (a requirement document generated directly from the initial intent), and Clarified Doc. (a requirement document generated after interactive clarification).
Singapore Management University, Singapore · Monash University Malaysia, Malaysia · Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea +5