Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Authors: Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong
Organizations: National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · Beihang University, China
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.
Figures & tables
Figure 1. Overview of the Req2Test-Bench task. Each instance provides a requirement composed of enclosing Feature context and a target Scenario , together with designated target interfaces and project context exposed through either a target-hidden or an all-hidden view. Both views withhold the target implementation. Target-hidden retains other permitted production implementations, whereas all-hidden withholds them. An LLM agent generates a complete, potentially multi-file Test Implementation . After generation, an evaluator runs the test against the complete production implementation, recording staged functional outcomes and comparing its production-API calls, code coverage, and mutant kills with those of the ground-truth Test Implementation.
Resource
Source
Granularity
Evaluation Criteria
Hierarchy
Size
#Pairs
#PLs
E2EDev ( Liu et al., 2026 )
Synthetic
Scenario ↔ Test Impl.
N/A
N/A
703
1
GivenWhenThen ( Júnior and Montandon, 2026 )
Real
Feature ↔ Test Files
N/A
N/A
2,289
3
AutoGlue ( Shi et al., 2026 )
Real
Step ↔ Method
Ref.-based
N/A
1,307
1
DevBench ( Li et al., 2024 )
Manual
Project ↔ Test Files
Prog.-based
N/A
22
4
Req2Test-Bench
Real
Scenario ↔ Test Impl.
Prog. & Ref.-based
\ding 51
5,086
5
Table 1. Comparison of resources pairing requirement-like inputs with ground-truth test code. “Source” indicates whether an artifact was preserved from the subject project. “Hierarchy” indicates test levels and explicit requirement relations.
Figure 2. BDD-grounded meta-model of Req2Test-Bench . A Requirement pairs an enclosing Feature with a target Scenario. A Feature is illustrated by Scenario s. Each Scenario contains an ordered sequence of Steps matched to executable Step Definitions and is automated by one complete, potentially multi-file Test Implementation. A Test Implementation contains its matched Step Definitions and their helper code. Dependency and variant relations may connect Features or Scenarios.
Statistic
Total
Benchmark
Projects
8
Programming languages
5
BDD frameworks
4
Instances
5,086
Features
1,415
Table 2. Statistics of Req2Test-Bench . Left: totals over all instances. Right: minimum, maximum, and mean per instance. C / I / S: component, integration, and system level. A ground-truth kill is an instance–mutant pair whose mutant the ground-truth test kills.
Instances
Mean per evaluated instance
Mutation
Project
Language / framework
Valid .
Eval.
Step defs.
Steps
Target I/F
Files
Lines
API
Mutants
Pairs
Kill ( % )
coddy-agent
Go, godog
282
274
835
5.8
3.3
2.5
1,357
8.6
20,243
227,886
24.8
Algorand SDK
Java, Cucumber
117
111
188
4.1
6.2
2.2
542
7.5
7,766
12,853
58.4
Kapua
Java, Cucumber
605
330
415
6.3
9.5
1.5
2,286
18.9
48,911
33,000
41.8
CoreShop
PHP, Behat
422
334
371
16.1
3.9
13.8
2,631
44.0
10,205
171,174
28.2
PrestaShop
PHP, Behat
1,352
348
713
21.5
13.8
8.2
2,911
35.0
71,670
34,355
29.4
Table 3. Project-level breakdown of Req2Test-Bench : instances, per-instance measurements of the Requirement and its ground-truth Test Implementation, and mutation data. Means are over the evaluated instances.
End-to-end success (%)
Similarity to ground truth (%)
System
Model
Reasoning
View
Pass@1
KillComplete@1
API F1
Coverage J
Mutation R
Aider
DS
off
Target-hidden
9.7
4.5
6.6
8.0
8.5
All-hidden
9.7
4.4
6.7
8.0
8.7
OpenHands
DS
off
Target-hidden
32.7
18.1
26.4
30.9
31.2
All-hidden
26.5
16.6
21.2
25.2
25.3
Claude Code
DS
high
Target-hidden
80.5
34.4
62.0
73.2
73.3
Table 4. Overall effectiveness: means over eight projects. A non-passing test counts as 0 in similarity columns.
Figure 3. Pass@1 (bar height) and KillComplete@1 (dark part), in percent, per project, view, and configuration.
Figure 4. An illustrative example from python-docx , sty-latent-props.feature:36 (target-hidden view). Left: the agent’s input and its test result. Right: the ground-truth test and the test generated by Codex/Sol. Yellow: what the Requirement leaves abstract and how the ground truth fills it; red: where the generated test constructs the object itself.
Aider
OpenHands
Claude Code
Codex CLI
DS, off
DS, off
DS, high
Sonnet, high
DS, off
DS, high
Luna, high
Sol, high
$ per generation
0.061
0.187
0.158
0.680
0.121
0.236
0.039
1.16
$ per kill-complete test
1.56
1.21
0.49
1.99
0.55
0.74
0.15
3.92
Time (minutes)
2.0
5.7
6.2
2.7
4.1
8.7
3.2
3.1
LLM calls
2.0
137.8
88.9
22.3
96.6
102.4
23.6
16.8
Table 5. Cost and resource usage per generation in the target-hidden view.
Figure 5. KillComplete@1 (filled) and Pass@1 (hollow) of the configurations, ordered by their cost per generation. The thick line joins the Pareto frontier of the point estimates, i.e., the configurations whose KillComplete@1 no other reaches at the same or lower cost.
Figure 6. (a) Where generations end, in percent of each configuration’s generations (project means over both views). Black ticks mark the end of the passing tests. (b) Why: failure modes of failing generations and kill-incomplete tests, in percent.
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.
Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
Existing REST API testing tools are typically evaluated using code coverage and crash-based fault metrics. However, recent LLM-based approaches increasingly generate tests from NL requirements to validate functional behaviour, making traditional metrics weak proxies for whether generated tests validate intended behaviour. To address this gap, we present RESTestBench, a benchmark comprising three REST services paired with manually verified NL requirements in both precise and vague variants, enabling controlled and reproducible evaluation of requirement-based test generation. RESTestBench further introduces a requirements-based mutation testing metric that measures the fault-detection effectiveness of a generated test case with respect to a specific requirement, extending the property-based approach of Bartocci et al. . Using RESTestBench, we evaluate two approaches across multiple state-of-the-art LLMs: (i) non-refinement-based generation, and (ii) refinement-based generation guided by interaction with the running SUT. In the refinement experiments, RESTestBench assesses how exposure to the actual implementation, valid or mutated, affects test effectiveness. Our results show that test effectiveness drops considerably when the generator interacts with faulty or mutated code, especially for vague requirements, sometimes negating the benefit of refinement and indicating that incorporating actual SUT behaviour is unnecessary when requirement detail is high.
Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1
Benchmarks are widely used to evaluate task completion by Large Language Models (LLMs), but this approach has accumulated construction-validity problems, and a passing score may not show whether the requested task was delivered. We study both problems. In a controlled code-as-spec setup, two production Copilot CLI agents (claude-opus-4.7, gpt-5.5) re-implement a React Fluent-UI data table in Angular as a reusable library under a hidden 222-test Playwright oracle across 18 runs and three oracle-availability conditions. Alongside the score, we run a mechanical library audit and check each verdict with a no-op ablation. Without the oracle, the library is present but unfinished, revealed by scores. With the oracle in the loop, the score reaches near-perfect, but from a demo holding the tested behavior directly, the library left dead or absent. We call this building to the test; the broader disposition behind both we call validation self-awareness. The agent does not, on its own, validate what it ships as a user would. Prevalence remains an open question across other agents, signals, and model families. Beyond benchmark scores, dispositions like validation self-awareness merit research attention.