Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications

Tzafrir Rehan

cs.SEMar 9, 2026

Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications

Authors: Tzafrir Rehan

Organizations: Fiverr Labs

Abstract

We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at https://github.com/f-labs-io/tdad-paper-code.

Figures & tables

Figure 1: TDAD overview. Blue boxes are LLM coding agents; gray boxes are deterministic measurement or infrastructure. TestSmith, PromptSmith, and the Compiled Agent form the compilation pipeline (left). MutationSmith operates post-compilation for evaluation only (right). Spec evolution (bottom) measures regression safety across versions.

Figure 2: Semantic mutation testing pipeline. Mutants that fail activation after 5 attempts are excluded as non-activating, analogous to filtering likely-equivalent mutants in traditional frameworks.

Metric	Definition	Purpose
VPR	Visible Pass Rate	Compilation
HPR	Hidden Pass Rate	Generalization
MS	Mutation Score	Test quality
SURS	Spec Update Regr.	Regression safety
RPR †	Reliability Pass Rate	Stochasticity
† Three trials provide variance estimates; formal RPR ( $N$ =10–50

Table 1: TDAD Evaluation Metrics

Spec	Key Challenges
SupportOps	Auth before action, PII refusal, plan eligibility, escalation triggers
DataInsights	SQL grounding, numeric precision, ambiguity detection, cost-aware queries
IncidentRunbook	Evidence-first ordering, severity routing, runbook compliance
ExpenseGuard	Spending caps, FX conversion, receipt requirements, manager approval

Table 2: SpecSuite-Core Specifications

Median test counts across 3 trials; varies $\pm$ 10–30% due to stochastic generation.
Spec	Nodes	V1 Vis.	V1 Hid.	V2 Vis.	V2 Hid.	Mut.
SupportOps	12	47	45	53	43	7
DataInsights	10	34	42	53	43	6
IncidentRunbook	14	39	42	42	45	7
ExpenseGuard	11	47	45	46	42	7

Table 3: SpecSuite-Core Quantitative Depth

	Compile		HPR (%)		MS (%)		SURS
Spec	V1	V2	V1	V2	V1	V2	(%)
SupportOps	3/3	2/3	97.6 $\pm$ 2.3	62.5 $\pm$ 16.1	100	100	94.2 $\pm$ 5.1
DataInsights	3/3	2/3	96.6 $\pm$ 4.1	88.3 $\pm$ 3.7	94.4 $\pm$ 9.6	100	98.7 $\pm$ 1.9
IncidentRunbook	2/3	2/3	95.2 $\pm$ 0.3	80.3 $\pm$ 13.1	85.7	100	97.3 $\pm$ 3.7
ExpenseGuard	3/3	1/3	99.2 $\pm$ 1.3	81.8 †	100	100 †	100 †
Aggregate	11/12	7/12	97.3 $\pm$ 2.6	77.7 $\pm$ 13.9	95.9 $\pm$ 7.1	100	97.2 $\pm$ 3.5

Table 4: TDAD Pipeline Results Across SpecSuite-Core (mean

\pm

std over successful runs)

	Cost ($)		Iterations
Spec	V1	V2	V1	V2
SupportOps	2.19 $\pm$ 1.06	2.02 $\pm$ 0.25	3.3 $\pm$ 0.6	2.5 $\pm$ 0.7
DataInsights	2.59 $\pm$ 1.38	2.12 $\pm$ 0.93	3.3 $\pm$ 0.6	2.5 $\pm$ 0.7
IncidentRunbook	1.55 $\pm$ 0.12	4.23 $\pm$ 3.21	2.5 $\pm$ 0.7	3.5 $\pm$ 2.1
ExpenseGuard	2.68 $\pm$ 0.83	2.91 †	3.3 $\pm$ 1.2	3.0 †
Average	2.32 $\pm$ 0.96	2.81 $\pm$ 1.71	3.1 $\pm$ 0.8	2.9 $\pm$ 1.1

Table 5: Pipeline Cost (USD) and Iterations (mean

\pm

std, successful runs)

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Spec	Mutation	V1	V2
SupportOps	SKIP_AUTH_GATE	K	K
	LEAK_PII_DIRECT_REQ	K	K
	SKIP_CONFIRM_CANCEL	K	K
	ALWAYS_CREATE_TICKET	K	K
DataInsights	ANSWER_WITHOUT_SQL	K	K
	HALLUCINATE_NUMBERS	S ∗	K

Appendix

Table 6: Selected Mutation Testing Results (K=Killed, S=Survived). Each spec has 5–7 total intents; the 4 most representative per spec are shown.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 277 papers

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis
Mar 18, 2026Pepe Alonso, Sergio Yovine, Victor A. BrabermanAI Coding Agents AI Agent Reliability
Universidad ORT Uruguay · DC, UBA, Argentina
GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing
Oct 2, 2026Masahiro KatoAutomated Software Testing AI Coding Agents
Mizuho-DL Financial Technology Co., Ltd.
Willful Disobedience: Automatically Detecting Failures in Agentic Traces
Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent Evaluation AI Agent Auditing
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

Mar 18, 2026J/K move · Enter open · S save

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis

Pepe Alonso, Sergio Yovine, Victor A. Braberman

Universidad ORT Uruguay · DC, UBA, Argentina

AI Coding Agents AI Agent Reliability Automated Software Testing

Open paper AskPDF arXiv

AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD.

Abstract

Tool-Using Agents

Explore similar work

TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis

GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing

Willful Disobedience: Automatically Detecting Failures in Agentic Traces