Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Authors: Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong
Organizations: National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · Beihang University, China
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.
Figures & tables
Figure 1. Overview of the Req2Test-Bench task. Each instance provides a requirement composed of enclosing Feature context and a target Scenario , together with designated target interfaces and project context exposed through either a target-hidden or an all-hidden view. Both views withhold the target implementation. Target-hidden retains other permitted production implementations, whereas all-hidden withholds them. An LLM agent generates a complete, potentially multi-file Test Implementation . After generation, an evaluator runs the test against the complete production implementation, recording staged functional outcomes and comparing its production-API calls, code coverage, and mutant kills with those of the ground-truth Test Implementation.
Resource
Source
Granularity
Evaluation Criteria
Hierarchy
Size
#Pairs
#PLs
E2EDev ( Liu et al., 2026 )
Synthetic
Scenario ↔ Test Impl.
N/A
N/A
703
1
GivenWhenThen ( Júnior and Montandon, 2026 )
Real
Feature ↔ Test Files
N/A
N/A
2,289
3
AutoGlue ( Shi et al., 2026 )
Real
Step ↔ Method
Ref.-based
N/A
1,307
1
DevBench ( Li et al., 2024 )
Manual
Project ↔ Test Files
Prog.-based
N/A
22
4
Req2Test-Bench
Real
Scenario ↔ Test Impl.
Prog. & Ref.-based
\ding 51
5,086
5
Table 1. Comparison of resources pairing requirement-like inputs with ground-truth test code. “Source” indicates whether an artifact was preserved from the subject project. “Hierarchy” indicates test levels and explicit requirement relations.
Figure 2. BDD-grounded meta-model of Req2Test-Bench . A Requirement pairs an enclosing Feature with a target Scenario. A Feature is illustrated by Scenario s. Each Scenario contains an ordered sequence of Steps matched to executable Step Definitions and is automated by one complete, potentially multi-file Test Implementation. A Test Implementation contains its matched Step Definitions and their helper code. Dependency and variant relations may connect Features or Scenarios.
Statistic
Total
Benchmark
Projects
8
Programming languages
5
BDD frameworks
4
Instances
5,086
Features
1,415
Table 2. Statistics of Req2Test-Bench . Left: totals over all instances. Right: minimum, maximum, and mean per instance. C / I / S: component, integration, and system level. A ground-truth kill is an instance–mutant pair whose mutant the ground-truth test kills.
Instances
Mean per evaluated instance
Mutation
Project
Language / framework
Valid .
Eval.
Step defs.
Steps
Target I/F
Files
Lines
API
Mutants
Pairs
Kill ( % )
coddy-agent
Go, godog
282
274
835
5.8
3.3
2.5
1,357
8.6
20,243
227,886
24.8
Algorand SDK
Java, Cucumber
117
111
188
4.1
6.2
2.2
542
7.5
7,766
12,853
58.4
Kapua
Java, Cucumber
605
330
415
6.3
9.5
1.5
2,286
18.9
48,911
33,000
41.8
CoreShop
PHP, Behat
422
334
371
16.1
3.9
13.8
2,631
44.0
10,205
171,174
28.2
PrestaShop
PHP, Behat
1,352
348
713
21.5
13.8
8.2
2,911
35.0
71,670
34,355
29.4
Table 3. Project-level breakdown of Req2Test-Bench : instances, per-instance measurements of the Requirement and its ground-truth Test Implementation, and mutation data. Means are over the evaluated instances.
End-to-end success (%)
Similarity to ground truth (%)
System
Model
Reasoning
View
Pass@1
KillComplete@1
API F1
Coverage J
Mutation R
Aider
DS
off
Target-hidden
9.7
4.5
6.6
8.0
8.5
All-hidden
9.7
4.4
6.7
8.0
8.7
OpenHands
DS
off
Target-hidden
32.7
18.1
26.4
30.9
31.2
All-hidden
26.5
16.6
21.2
25.2
25.3
Claude Code
DS
high
Target-hidden
80.5
34.4
62.0
73.2
73.3
Table 4. Overall effectiveness: means over eight projects. A non-passing test counts as 0 in similarity columns.
Figure 3. Pass@1 (bar height) and KillComplete@1 (dark part), in percent, per project, view, and configuration.
Figure 4. An illustrative example from python-docx , sty-latent-props.feature:36 (target-hidden view). Left: the agent’s input and its test result. Right: the ground-truth test and the test generated by Codex/Sol. Yellow: what the Requirement leaves abstract and how the ground truth fills it; red: where the generated test constructs the object itself.
Aider
OpenHands
Claude Code
Codex CLI
DS, off
DS, off
DS, high
Sonnet, high
DS, off
DS, high
Luna, high
Sol, high
$ per generation
0.061
0.187
0.158
0.680
0.121
0.236
0.039
1.16
$ per kill-complete test
1.56
1.21
0.49
1.99
0.55
0.74
0.15
3.92
Time (minutes)
2.0
5.7
6.2
2.7
4.1
8.7
3.2
3.1
LLM calls
2.0
137.8
88.9
22.3
96.6
102.4
23.6
16.8
Table 5. Cost and resource usage per generation in the target-hidden view.
Figure 5. KillComplete@1 (filled) and Pass@1 (hollow) of the configurations, ordered by their cost per generation. The thick line joins the Pareto frontier of the point estimates, i.e., the configurations whose KillComplete@1 no other reaches at the same or lower cost.
Figure 6. (a) Where generations end, in percent of each configuration’s generations (project means over both views). Black ticks mark the end of the passing tests. (b) Why: failure modes of failing generations and kill-incomplete tests, in percent.