GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing

Masahiro Kato

cs.SEOct 2, 2026

GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing

Authors: Masahiro Kato

Organizations: Mizuho-DL Financial Technology Co., Ltd.

Abstract

Test-driven development gives AI coding agents executable requirements for implementing software. Because these agents can adapt their implementations to the examples they observe, passing a predetermined collection of tests can leave substantial parts of the intended behavior unimplemented. We propose Generative Test-Driven Development (GTDD), a formulation of test-driven development in which a separate testing agent generates new inputs after each candidate implementation is fixed, using a human-specified behavioral contract and the feedback from earlier rounds. A trusted evaluator checks these inputs, returns reduced counterexamples to the coding agent, and saves them for regression testing, so development continually confronts failures beyond the initial examples. We characterize the evidence that this process provides through a finite-population analysis of false acceptance under adaptive candidate selection. The resulting bounds quantify how test visibility and repeated feedback affect acceptance, and show that fresh random audits after candidate commitment control false acceptance across development rounds. In a paired experiment on a stateful key-value store, both policies that regenerated tests during development ended with lower mean failure rates than the policy whose tests were generated once by the same language model, and giving the tester the candidate's source produced no detectable additional improvement. Further conditions requesting equal numbers of tests did not isolate any single feature of the policies as the source of this difference. GTDD combines this adaptive development feedback with established regression tests and an independent acceptance rule.

Figures & tables

Panel of Figure 1 — Figure 1: Order of a development round. The test-generation policy supplies inputs, the feedback rule returns at most $k$ reduced failures to the next coding round, and the acceptance rule determines whether the candidate is accepted. Population evaluation follows the recorded acceptance decision and is never reported to either agent.

Test selection	Sufficient size condition	New cases per round
Known fixed tests	$n\geq N-\lfloor\rho N\rfloor$	$0$ after setup
Hidden fixed tests	$H_{T}\,Q(n)\leq\alpha$	$0$ after setup
Uniform generation	$T\,Q(m)\leq\alpha$	$m$
Fresh audit	$T\,Q(n)\leq\alpha$	$n$

Table 1: Sufficient acceptance conditions over a common horizon. The uniform-generation row assumes the sampling rule in Proposition 4.8 . Case counts refer to new samples and exclude repeated execution of regression tests.

Guarantee target	Exact $n$	Exponential $n$	$q_{500,0.05}(n)$
One candidate	$54$	$59$	$0.0472088$
Repository over $32$ rounds	$108$	$126$	$0.00148353$
$160$ family conclusions	$131$	$158$	$0.000291773$

Table 2: Smallest exact audit sizes and exponential sufficient sizes. The repository row covers

8

trajectories of

4

rounds; the family row additionally covers

5

separate family conclusions per round.

Hidden test-set size $n$	Union lower bound	Exact success
$54$	$0.9989$	$0.9989$
$100$	$0.9504$	$0.9511$
$104$	$0.9385$	$0.9395$
$108$	$0.9245$	$0.9260$
$131$	$0.7963$	$0.8082$
$158$	$0.5199$	$0.5876$

Table 3: Success of the three-region probe construction with

N=500

m=15

, and

k=6

. The probabilities describe the explicit casewise adversary after all four prescribed rounds.

Failures	Risk	Pass at $n=54$	Pass at $n=108$	Pass at $n=131$
$0$	$0.000$	$1.000000$	$1.000000$	$1.000000$
$1$	$0.002$	$0.892000$	$0.784000$	$0.738000$
$5$	$0.010$	$0.563334$	$0.294558$	$0.217358$
$10$	$0.020$	$0.315393$	$0.085553$	$0.046396$
$25$	$0.050$	$0.053264$	$0.001920$	$0.000403$
$26$	$0.052$	$0.047209$	$0.001484$	$0.000292$

Table 4: Exact probabilities that one family’s zero-failure audit passes. The count

26

is the first above tolerance.

Condition	Round 1	Round 2	Round 3	Round 4	New counterexamples in round 3
Public only	$0.548$	$0.346$	$0.342$	$0.342$	—
Static property	$0.548$	$0.396$	$0.309$	$0.298$	$4.93$
Static LLM	$0.548$	$0.382$	$0.363$	$0.405$	$3.40$
Dynamic blind	$0.548$	$0.360$	$0.357$	$0.279$	$5.77$
Dynamic informed	$0.548$	$0.441$	$0.316$	$0.274$	$5.37$

Table 5: Mean evaluation failure rate after each coding round over

30

blocks, and the mean number of counterexamples newly disclosed after round three, which the last repair round received. Round one is the shared first-round module.

Contrast (first minus second)	Difference	95% interval	$p$	Lower and higher
Dynamic informed $-$ static LLM	$-0.131$	$[-0.228,-0.050]$	$0.0015$	$19,7$
Dynamic informed $-$ dynamic blind	$-0.005$	$[-0.044,+0.034]$	$0.810$	$12,15$
Dynamic blind $-$ static LLM	$-0.126$	$[-0.223,-0.044]$	$0.0012$	$20,8$
Dynamic informed $-$ static property	$-0.024$	$[-0.097,+0.043]$	$0.534$	$13,13$
Static property $-$ static LLM	$-0.107$	$[-0.205,-0.018]$	$0.029$	$20,7$
Dynamic informed $-$ public only	$-0.067$	$[-0.103,-0.032]$	$0.0007$	$20,8$

Table 6: Within-block differences of the failure rate after round four. The last column counts the blocks in which the first condition was lower and higher; the remaining blocks are ties. The first row is the primary comparison.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Round 1	Round 2	Round 3	Round 4	Valid tests
Static LLM	$0.548$	$0.382$	$0.363$	$0.405$	$9{,}386$
Static LLM 600	$0.548$	$0.377$	$0.358$	$0.355$	$14{,}291$
Staged 600	$0.548$	$0.352$	$0.346$	$0.311$	$14{,}291$
Regenerated without history	$0.548$	$0.347$	$0.304$	$0.297$	$15{,}381$
Dynamic blind	$0.548$	$0.360$	$0.357$	$0.279$	$13{,}816$
Blind rerun	$0.548$	$0.396$	$0.346$	$0.308$	$12{,}394$

Appendix

Table 7: Mean evaluation failure rate after each coding round over

30

blocks for Static LLM, the four added conditions, and Dynamic blind, with the number of valid generated traces each condition received in total.

Contrast (first minus second)	Difference	95% interval	$p$	Lower and higher
Staged 600 $-$ static LLM 600	$-0.044$	$[-0.132,+0.033]$	$0.326$	$15,9$
Regenerated without history $-$ staged 600	$-0.014$	$[-0.083,+0.054]$	$0.694$	$17,12$
Blind rerun $-$ regenerated without history	$+0.012$	$[-0.040,+0.078]$	$0.794$	$15,13$
Static LLM 600 $-$ static LLM	$-0.050$	$[-0.163,+0.052]$	$0.378$	$16,11$
Blind rerun $-$ dynamic blind	$+0.029$	$[-0.027,+0.097]$	$0.446$	$11,17$
Blind rerun $-$ static LLM 600	$-0.047$	$[-0.121,+0.037]$	$0.282$	$17,12$

Appendix

Table 8: Within-block differences of the failure rate after round four involving the added conditions. The last column counts the blocks in which the first condition was lower and higher.

Contrast (first minus second)	Mean	Median	Trimmed mean	Sign test $p$
Dynamic informed $-$ static LLM	$-0.131$	$-0.044$	$-0.076$	$0.029$
Dynamic informed $-$ dynamic blind	$-0.005$	$+0.001$	$-0.006$	$0.701$
Dynamic blind $-$ static LLM	$-0.126$	$-0.023$	$-0.061$	$0.036$
Dynamic informed $-$ static property	$-0.024$	$+0.000$	$-0.009$	$1.000$
Static property $-$ static LLM	$-0.107$	$-0.065$	$-0.086$	$0.019$
Dynamic informed $-$ public only	$-0.067$	$-0.029$	$-0.061$	$0.036$

Appendix

Table 9: Summaries of the

30

within-block differences after round four. The trimmed mean removes the

3

largest and

3

smallest differences, and the exact sign test drops ties.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 131 papers

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis
Mar 18, 2026Pepe Alonso, Sergio Yovine, Victor A. BrabermanAI Coding Agents AI Agent Reliability
Universidad ORT Uruguay · DC, UBA, Argentina
Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications
Mar 9, 2026Tzafrir RehanTool-Using Agents AI Agent Evaluation
Fiverr Labs
TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation
Sep 29, 2025Yiran Hu, Shanchao Liang, Nan Jiang +2Code Generation Agentic Code Generation
Purdue University

Mar 18, 2026J/K move · Enter open · S save

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis

Pepe Alonso, Sergio Yovine, Victor A. Braberman

Universidad ORT Uruguay · DC, UBA, Argentina

AI Coding Agents AI Agent Reliability Automated Software Testing

Open paper AskPDF arXiv

AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD.

Abstract

Automated Software Testing

Explore similar work

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis

Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications

TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation