Test-driven development gives AI coding agents executable requirements for implementing software. Because these agents can adapt their implementations to the examples they observe, passing a predetermined collection of tests can leave substantial parts of the intended behavior unimplemented. We propose Generative Test-Driven Development (GTDD), a formulation of test-driven development in which a separate testing agent generates new inputs after each candidate implementation is fixed, using a human-specified behavioral contract and the feedback from earlier rounds. A trusted evaluator checks these inputs, returns reduced counterexamples to the coding agent, and saves them for regression testing, so development continually confronts failures beyond the initial examples. We characterize the evidence that this process provides through a finite-population analysis of false acceptance under adaptive candidate selection. The resulting bounds quantify how test visibility and repeated feedback affect acceptance, and show that fresh random audits after candidate commitment control false acceptance across development rounds. In a paired experiment on a stateful key-value store, both policies that regenerated tests during development ended with lower mean failure rates than the policy whose tests were generated once by the same language model, and giving the tester the candidate's source produced no detectable additional improvement. Further conditions requesting equal numbers of tests did not isolate any single feature of the policies as the source of this difference. GTDD combines this adaptive development feedback with established regression tests and an independent acceptance rule.
Figures & tables
Figure 1: Order of a development round. The test-generation policy supplies inputs, the feedback rule returns at most k reduced failures to the next coding round, and the acceptance rule determines whether the candidate is accepted. Population evaluation follows the recorded acceptance decision and is never reported to either agent.
Test selection
Sufficient size condition
New cases per round
Known fixed tests
n≥N−⌊ρN⌋
0 after setup
Hidden fixed tests
HTQ(n)≤α
0 after setup
Uniform generation
TQ(m)≤α
m
Fresh audit
TQ(n)≤α
n
Table 1: Sufficient acceptance conditions over a common horizon. The uniform-generation row assumes the sampling rule in Proposition 4.8 . Case counts refer to new samples and exclude repeated execution of regression tests.
Guarantee target
Exact n
Exponential n
q500,0.05(n)
One candidate
54
59
0.0472088
Repository over 32 rounds
108
126
0.00148353
160 family conclusions
131
158
0.000291773
Table 2: Smallest exact audit sizes and exponential sufficient sizes. The repository row covers 8 trajectories of 4 rounds; the family row additionally covers 5 separate family conclusions per round.
Hidden test-set size n
Union lower bound
Exact success
54
0.9989
0.9989
100
0.9504
0.9511
104
0.9385
0.9395
108
0.9245
0.9260
131
0.7963
0.8082
158
0.5199
0.5876
Table 3: Success of the three-region probe construction with N=500 , m=15 , and k=6 . The probabilities describe the explicit casewise adversary after all four prescribed rounds.
Failures
Risk
Pass at n=54
Pass at n=108
Pass at n=131
0
0.000
1.000000
1.000000
1.000000
1
0.002
0.892000
0.784000
0.738000
5
0.010
0.563334
0.294558
0.217358
10
0.020
0.315393
0.085553
0.046396
25
0.050
0.053264
0.001920
0.000403
26
0.052
0.047209
0.001484
0.000292
Table 4: Exact probabilities that one family’s zero-failure audit passes. The count 26 is the first above tolerance.
Condition
Round 1
Round 2
Round 3
Round 4
New counterexamples in round 3
Public only
0.548
0.346
0.342
0.342
—
Static property
0.548
0.396
0.309
0.298
4.93
Static LLM
0.548
0.382
0.363
0.405
3.40
Dynamic blind
0.548
0.360
0.357
0.279
5.77
Dynamic informed
0.548
0.441
0.316
0.274
5.37
Table 5: Mean evaluation failure rate after each coding round over 30 blocks, and the mean number of counterexamples newly disclosed after round three, which the last repair round received. Round one is the shared first-round module.
Contrast (first minus second)
Difference
95% interval
p
Lower and higher
Dynamic informed − static LLM
−0.131
[−0.228,−0.050]
0.0015
19,7
Dynamic informed − dynamic blind
−0.005
[−0.044,+0.034]
0.810
12,15
Dynamic blind − static LLM
−0.126
[−0.223,−0.044]
0.0012
20,8
Dynamic informed − static property
−0.024
[−0.097,+0.043]
0.534
13,13
Static property − static LLM
−0.107
[−0.205,−0.018]
0.029
20,7
Dynamic informed − public only
−0.067
[−0.103,−0.032]
0.0007
20,8
Table 6: Within-block differences of the failure rate after round four. The last column counts the blocks in which the first condition was lower and higher; the remaining blocks are ties. The first row is the primary comparison.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Round 1
Round 2
Round 3
Round 4
Valid tests
Static LLM
0.548
0.382
0.363
0.405
9,386
Static LLM 600
0.548
0.377
0.358
0.355
14,291
Staged 600
0.548
0.352
0.346
0.311
14,291
Regenerated without history
0.548
0.347
0.304
0.297
15,381
Dynamic blind
0.548
0.360
0.357
0.279
13,816
Blind rerun
0.548
0.396
0.346
0.308
12,394
Appendix
Table 7: Mean evaluation failure rate after each coding round over 30 blocks for Static LLM, the four added conditions, and Dynamic blind, with the number of valid generated traces each condition received in total.
Contrast (first minus second)
Difference
95% interval
p
Lower and higher
Staged 600 − static LLM 600
−0.044
[−0.132,+0.033]
0.326
15,9
Regenerated without history − staged 600
−0.014
[−0.083,+0.054]
0.694
17,12
Blind rerun − regenerated without history
+0.012
[−0.040,+0.078]
0.794
15,13
Static LLM 600 − static LLM
−0.050
[−0.163,+0.052]
0.378
16,11
Blind rerun − dynamic blind
+0.029
[−0.027,+0.097]
0.446
11,17
Blind rerun − static LLM 600
−0.047
[−0.121,+0.037]
0.282
17,12
Appendix
Table 8: Within-block differences of the failure rate after round four involving the added conditions. The last column counts the blocks in which the first condition was lower and higher.
Contrast (first minus second)
Mean
Median
Trimmed mean
Sign test p
Dynamic informed − static LLM
−0.131
−0.044
−0.076
0.029
Dynamic informed − dynamic blind
−0.005
+0.001
−0.006
0.701
Dynamic blind − static LLM
−0.126
−0.023
−0.061
0.036
Dynamic informed − static property
−0.024
+0.000
−0.009
1.000
Static property − static LLM
−0.107
−0.065
−0.086
0.019
Dynamic informed − public only
−0.067
−0.029
−0.061
0.036
Appendix
Table 9: Summaries of the 30 within-block differences after round four. The trimmed mean removes the 3 largest and 3 smallest differences, and the exact sign test drops ties.
AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD.
We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at https://github.com/f-labs-io/tdad-paper-code.
Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation. With recent advances in Large Language Models (LLMs), developers can shift from manually writing the code to defining tests as executable specifications and delegating code synthesis to AI agents. However, enabling repository-level TDD under developer-written tests is challenging, requiring: (1) specification enhancement: identifying a concise yet representative test subset from large suites with rich task semantics; (2) retrieval augmentation: using tests to guide reasoning and context retrieval; and (3) test-driven refinement: interpreting test feedback for iterative improvement. We propose TENET, an agentic framework for repository-level code generation under the TDD paradigm. TENET includes: (1) a test harness mechanism that selects a concise test suite to maximize diversity of the target usage scenarios; (2) a tailored agent toolset for efficient retrieval and debugging; and (3) a reflection-based refinement workflow that iteratively analyzes failures and updates implementations. TENET consistently outperforms the strongest baselines across backbones, achieving 69.08% and 81.77% Pass@1 on RepoCod and RepoEval with Claude Sonnet 4, improving by 9.49 and 2.17 percentage points, respectively. Additionally, we present the first systematic study of how test suite characteristics influence LLM agent performance in TDD settings.