Organizations: National College for Excellent Engineers, Beihang University · School of Computer Science and Engineering, Beihang University · Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University · School of Software, Beihang University
Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect. In this paper, we introduce TDD-Agent, which operationalizes the test-driven development paradigm for code generation. TDD-Agent first prompts the model to generate executable tests, encouraging it to clarify expected behaviors before implementation, and then performs iterative dual-track refinement over both the generated code and tests using execution feedback. We first isolate the effect of test-first reasoning through a prompt variant TDD-prompt on LiveCodeBench, where it consistently improves upon reasoning-based prompting baselines. Building on this finding, we evaluate the full TDD-Agent framework on RepoEval, a repository-level benchmark, and show that it consistently outperforms retrieval-based and agent-based baselines. Additional analyses show that iterative refinement improves not only code correctness but also the effectiveness of the generated tests, yielding higher pass rates, coverage, and mutation scores, suggesting that tests can serve as evolving reasoning artifacts rather than fixed validators. Our source code is available at https://anonymous.4open.science/r/TDD-Agent-Framework-6370/.
Figures & tables
Figure 1: An example demonstrating that formulating tests constitutes a form of reasoning.
Figure 2: Overview of the TDD-Agent. (1) The LLM first designs and generates unit tests for the target function. (2) The LLM generates the complete implementation of the target function, and through execution and refinement, iteratively refines the tests and code.
Method
GPT
DeepSeek
Qwen
one-shot
53.08
45.40
39.60
CoT-prompt
67.41
67.46
42.99
SCoT-prompt
66.83
64.20
40.80
Self-Planning
68.17
66.70
42.68
ICoT-prompt
68.48
62.23
41.88
TDD-prompt
70.04
67.86
44.87
Table 1: Pass@1 (%) comparison on LiveCodeBench across three models. Bold and underline indicate the best and second-best performance.
Method
GPT
DeepSeek
Qwen
In-File
44.84
39.34
32.75
RAG
50.55
46.15
35.16
RepoCoder
55.16
50.55
41.10
mini-SWE-agent
61.31
84.18
52.97
TDD-Agent-iter5
77.36
88.13
58.46
TDD-Agent-iter10
78.24
90.77
59.34
Table 2: Performance comparison on RepoEval across three models. Numbers are presented in percentage (%). Bold and underline indicate the best and second-best among all compared methods.
Model
Method
Prompt
Completion
Calls
GPT
mini
115.76
1.32
11.95
TDD-iter5
120.48
1.62
18.62
TDD-iter10
155.61
1.71
22.92
DS
mini
390.10
6.38
31.61
TDD-iter5
361.64
9.71
29.39
TDD-iter10
403.01
10.66
31.49
Table 3: Average token usage (in thousands) and average LLM calls across different models on mini-SWE-agent and TDD-Agent.
Figure 3: The pass rate improvement across iterations on three models.
Figure 4: Comparison between End Rate and Match Rate across different iteration rounds on three models. Match Rate represents the proportion of generated code that successfully passes the generated tests. End Rate indicates the proportion of tasks where the model autonomously decides to terminate.
Figure 5: Pass rate, coverage rate, and mutation score of generated tests over iterations for three models.
Variant
GPT
DeepSeek
Qwen
Vanilla
68.35
73.63
52.97
Reflect
72.09
81.53
54.06
Single-track
70.11
79.56
57.36
TDD-Agent-iter1
69.89
76.48
54.51
TDD-Agent
78.24
90.77
59.34
Table 4: Ablation study on RepoEval. Results are reported as pass rate (%). Bold and underline indicate the best and second-best performance for each model.
Figure 6: Distribution of matched and unmatched failures across three backbone models.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Parameter
Value
GPT-5-mini
temperature
1.0
max_tokens
4096
reasoning_effort
minimal
verbosity
low
DeepSeek-V3.2
temperature
1.0
max_tokens
4096
Appendix
Table 5: Generation parameter settings used in experiments for all models.
Method
GPT
DeepSeek
Qwen
TDD-prompt
70.04
67.86
44.87
w/o Test Generation
68.39
67.28
43.84
Appendix
Table 6: Ablation study results on LiveCodeBench across three models.
Method
Repo
All
1
2
3
4
5
6
7
8
GPT-5-mini
In-File
63.89
67.39
61.19
39.73
28.12
50.00
40.91
19.05
44.84
RAG
69.44
76.09
59.70
45.21
32.81
50.00
45.45
40.48
50.55
RepoCoder
72.22
78.26
70.15
50.68
39.06
50.00
40.91
42.86
55.16
mini-SWE-agent
80.56
69.57
68.66
58.22
46.88
65.63
63.64
52.38
61.31
Appendix
Table 7: Performance comparison on RepoEval. Numbers are presented in percentage (%), with the best performance highlighted in bold.
ID
Link
Count
1
leopard-ai/betty
36
2
CarperAI/trlx
46
3
lucidrains/imagen-pytorch
67
4
deepmind/tracr
146
5
google/lightweight_mmm
64
6
amazon-science/inspection
32
Appendix
Table 8: Detailed information of the GitHub repositories used for RepoEval.
Iter
GPT-5-mini
DeepSeek-V3.2
Qwen3-Coder-30B-A3B-Instruct
Prompt
Completion
Prompt
Completion
Prompt
Completion
Cached
Total
1
56674
1111
158337
3637
59054
1685
53510
7230
2
80363
1371
236649
5961
85475
2342
79185
8633
3
98505
1516
295616
7895
102012
2714
95369
9357
4
110656
1585
335806
9042
110371
2898
103563
9705
5
120478
1621
361637
9711
117065
3024
110155
9934
Appendix
Table 9: Average token usage of TDD-Agent across iterations for three models on RepoEval.
Figure 7: The tool usage statistics of TDD-Agent on three models.
Model
Method
Wall-clock Time(s)
Qwen
mini-SWE-agent
26.71
TDD-Agent-iter5
42.47
TDD-Agent-iter10
48.43
Appendix
Table 10: Average wall-clock time for mini-SWE-agent and TDD-Agent on locally deployed Qwen for one task.
Model
Credit Assignment Problem Rate
GPT
3.03 %
DeepSeek
4.76 %
Qwen
8.10 %
Appendix
Table 11: The Credit Assignment Problem Rate in failure cases across three models.
Figure 8: Generated-Tests Distribution in Matched Failures.
# System
You are an expert Python programmer. Given a programming problem, you must follow a strict thinking process before writing the final implementation.
# User
# Problem Description:
{description}
Appendix
Table 12: Prompt for TDD-prompt on LiveCodeBench.
# System
You are an expert Python programmer. Given a programming problem, you must follow a strict thinking process before writing the final implementation.
# User
# Problem Description:
{prompt}
Appendix
Table 13: Prompt for Ablation Study on LiveCodeBench.
# System
You’re a software engineer working inside an existing Python codebase rooted at /testbed.
Your task is to write test cases and complete the implementation of a target function whose implementation is currently missing. You can use these tools:
1. read_files
2. search_command
3. inspect_structure
4. list_files
Appendix
Table 14: Prompt for TDD-Agent on RepoEval.
# System
You’re a software engineer working inside an existing Python codebase rooted at /testbed.
Your task is to write test cases and complete the implementation of a target function whose implementation is currently missing. You can use these tools:
1. read_files
2. search_command
3. inspect_structure
4. list_files
Appendix
Table 15: Prompt for Ablation Study on RepoEval: Vanilla.
# System
You’re a software engineer working inside an existing Python codebase rooted at /testbed.
Your task is to write test cases and complete the implementation of a target function whose implementation is currently missing. You can use these tools:
1. read_files
2. search_command
3. inspect_structure
4. list_files
Appendix
Table 16: Prompt for Ablation Study on RepoEval: Reflect.
# System
You’re a software engineer working inside an existing Python codebase rooted at /testbed.
Your task is to write test cases and complete the implementation of a target function whose implementation is currently missing. You can use these tools:
1. read_files
2. search_command
3. inspect_structure
4. list_files
Appendix
Table 17: Prompt for Ablation Study on RepoEval: Single-track.
Problem Description:
You are given an integer array enemyEnergies denoting the energy values of various enemies.
You are also given an integer currentEnergy denoting the amount of energy you have initially.
You start with 0 points, and all the enemies are unmarked initially.
You can perform either of the following operations zero or multiple times to gain points:
Appendix
Table 18: Case Study: An Example Problem from LiveCodeBench and TDD-Prompt’s Final Answer.
Table 19: Case Study: A Response Example Generated by TDD-Prompt.
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China