Software is increasingly written by AI agents that produce an implementation from a natural language specification, a practice known as agentic coding. Code is now produced faster than it can be reviewed, and has been shown to exhibit its own characteristic failure patterns. Property-based testing (PBT), a long-standing technique in which invariant properties of a program are checked against many generated inputs, has recently been revisited with LLM agents authoring the properties, thereby removing the high-effort derivation of invariants that has long hindered its adoption. Existing methods, however, derive these properties from the implementation alone (Code-driven PBT). At the same time, a thorough specification such as a product requirements document has become routine within agentic coding, yet its role in testing the resulting code remains underexplored. We study the value of the specification as a source of properties for testing agent-written code. Our key insight is that the specification, unlike the implementation, records what the code was meant to do, and invariants derived from it are not biased by particular implementation choices. We propose Spec-Driven PBT, an agent that first expresses invariants in natural language from the specification and only then inspects the implementation to translate them into concrete tests. Across 25 applications from ViBench, Spec-Driven PBT finds a novel set of bugs that Code-driven PBT misses, while writing 2.4x fewer property tests. Though the two approaches surface a roughly equivalent number of bugs, there is little overlap: nearly 60% of the 169 total bugs are found by only one. Qualitatively, we find that the bugs unique to Spec-Driven PBT tend to violate documented product workflows whereas those to Code-driven PBT instead concentrate on implementation robustness, offering complementary value for bug discovery in agent-written code.
Figures & tables
Figure 1: Spec-Driven PBT (top) against the Code-driven PBT baseline (bottom). Spec-Driven PBT starts by deriving natural-language invariants from the specification alone, whereas Code-driven PBT derives properties directly from the implementation. Both methods then translate to Hypothesis tests, run pytest, and triage failures, with two feedback paths apiece.
Dataset
Method
Tests
Bug-bearing
Bugs (recall)
Unique
Bugs/test
ViBench
Spec-driven
566
34%
126
56
0.22
(25 PRDs)
Code-driven
1,384
13%
113
43
0.08
Found by both: 70 of 169 distinct bugs (Jaccard 41%)
PBTBench
Spec-driven
2,314
36%
321 (88%)
33
0.14
(100 problems)
Code-driven
3,135
25%
309 (85%)
21
0.10
Found by both: 288 of 342 distinct bugs (Jaccard 84%)
Table 1: Spec-Driven PBT (ours) against Code-driven PBT across two datasets. Test is a source-level test function or an exported Hypothesis state-machine case. Bugs means different things by dataset: ViBench carries no ground truth, so its figures are candidate bugs attributed by the blind reviewer. For PBTBench, these are injected bugs recovered under its strict fail-on-buggy, pass-on-correct (F → P) protocol, reported as recall over 365 injected bugs. Bug-bearing is the share of tests tied to ≥ 1 bug, by reviewer attribution on ViBench and by F → P confirmation on PBTBench.
Figure 2: Taxonomy of the candidate bugs found exclusively by each method. The inner ring gives abbreviated broad-class names; numbers beside the outer slices map the non-empty fine-grained classes to the legend. Percentages are computed within each method-exclusive set. Full definitions and counts appear in Appendix H .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Application
Method
Tests
Pass rate
Fail rate
Underlying bugs
collabrative kaban
Spec-Driven
23
39% (9)
61% (14)
6
Code-driven
31
71% (22)
29% (9)
5
creative community
Spec-Driven
15
33% (5)
67% (10)
6
Code-driven
44
80% (35)
20% (9)
4
energy audit
Spec-Driven
29
52% (15)
48% (14)
5
Code-driven
38
71% (27)
29% (11)
6
Appendix
Table 2: Per-application ViBench results, both methods, all 25 problems. Pass and fail rates use passed plus failed terminal pytest cases as the denominator, with counts in parentheses. “Underlying bugs” counts the candidate bugs the blind reviewer attributed to that suite. Table 1 gives the totals.
Figure 3: Candidate bugs against property tests authored, one point per generation cell. Stars mark each method’s median and the dashed guides are lines of constant yield.
Agent time (min)
Tokens (M)
API cost ($)
Method
Mean
Median
Max
Mean
Median
Max
Mean
Median
Max
Spec-Driven
31.1
31.2
50.5
9.45
8.48
17.50
8.22
7.83
12.82
Code-driven
29.9
30.1
54.0
8.94
7.33
21.79
7.69
6.60
16.47
Appendix
Table 3: Generation cost per cell on ViBench, 25 PRDs per method.
Figure 4: Final suite runtime per cell on a log scale, one point per generation cell. The solid rule is each method’s median, the dashed rule its mean.
Method
Tests
Bug-bearing
Bugs
Unique
Bugs/test
Spec-only
704
14.2%
78
41
0.11
Code-only
502
31.5%
117
80
0.23
Found by both: 37 of 158 distinct candidates (Jaccard 23.4%)
Appendix
Table 4: Controlled ViBench comparison. Tests are source-level functions; bug-bearing tests and candidate bugs are attributed by the pooled LLM reviewer, not ground truth.
Figure 5: Method-exclusive candidate bugs in the controlled comparison. Inner rings show broad classes; numbered outer slices map to fine classes in the legend. Percentages are within each arm.
Method
Run
Tests
Candidate bugs
Input handling
State/concurrency
Spec-Driven
0
566
131
36 (27%)
38 (29%)
1
519
115
30 (26%)
35 (30%)
2
538
123
32 (26%)
38 (31%)
Code-driven
0
1384
118
49 (42%)
23 (19%)
1
1417
130
61 (47%)
20 (15%)
2
1339
131
63 (48%)
22 (17%)
Appendix
Table 5: Tests, reviewer-attributed candidate bugs, and class counts across three runs per method. Class entries give counts (share of that run’s candidate bugs); state/concurrency combines state, lifecycle, data integrity, concurrency, and atomicity.
Figure 6: Bug-class composition across three runs per method, using the same sunburst styling and taxonomy as Figure 2 . “Concurrency” abbreviates “Concurrency and Atomicity” inside the smaller plots. Numbered outer slices map to fine classes in the shared legend. A candidate appearing in multiple runs is counted in each of those runs.
Broad category
Spec-Driven only ( n=56 )
Code-driven only ( n=43 )
Input Handling and Robustness
9 (16.1%)
21 (48.8%)
Domain Logic, Computation and Internal Contracts
15 (26.8%)
8 (18.6%)
State, Lifecycle and Data Integrity
19 (33.9%)
5 (11.6%)
Concurrency and Atomicity
7 (12.5%)
4 (9.3%)
Access Control, Identity and Untrusted Content
6 (10.7%)
5 (11.6%)
Appendix
Table 6: Distribution of method-exclusive ViBench candidate bugs across the five broad categories.
Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing: deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library's documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1-L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning.
Lucas Jing, Xinqi Wang, Liao Zhang +1
Tsinghua University · University of Washington · Liaoyuan Technology
As AI systems generate an ever-growing share of the world's code, formal verification offers a principled way to ensure that code is correct, and AI itself may be able to shoulder much of the verification burden. Yet we lack a clear picture of how well today's models and agents perform on verification tasks drawn from real-world software, since existing benchmarks rely largely on curated or synthetic problems, or focus on math rather than program verification. To close this gap, we present a benchmark for evaluating AI on real-world formal software verification tasks. We first scrape 7,413 distinct property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,623 of them (35%) into 9,415 Lean 4 specifications with sorry placeholders (3.5 formalizations/PBT). Translating PBTs to Lean specs is challenging: it requires modeling Python in Lean, inferring the property encoded in a PBT, and handling side effects. We describe an agentic pipeline for transpiling PBTs into Lean, evaluate coverage and quality metrics, and provide baselines for proof generation using several approaches. All code (scraper and agents) and data (PBTs and Lean specs) are open source.
Quinn Dougherty, Max von Hippel, Simon Henniger +2
Forall R&D, work completed at Galois Inc · work completed at Galois Inc · Some work completed at Benchify +2
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.
Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2