A Theoretical Analysis of Test-Driven Code Generation

Nicolas Menet; Michael Hersche; Andreas Krause; Abbas Rahimi

cs.SEFeb 5, 2026

A Theoretical Analysis of Test-Driven Code Generation

Authors: Nicolas Menet, Michael Hersche, Andreas Krause, Abbas Rahimi

Abstract

Code assistants are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant paradigms: code selection after generation using the execution environment, and code generation conditioned on environment feedback. First, we formalize several well-established selection heuristics as environment-aware estimators of code correctness. We theoretically prove that estimators based on fuzzy functional similarity add an inductive bias and strictly dominate estimators based on functional equivalence in terms of signal-to-noise ratio. Second, we frame backprompting as an in-context approximation of Thompson sampling. We derive a novel regret bound for reward functions with unobservable components, theoretically explaining why the effectiveness of backprompting is limited by the ambiguity of the informal task description (an irreducible regret). Using seven state-of-the-art open weight models, we corroborate these findings across BigCodeBenchHard, LeetCodeDataset, QiskitHumanEvalSim, and SWE-bench Verified. Our formalization also suggests how to improve task descriptions effectively, leading to an improved benchmark for LLM-assisted quantum programming, QiskitHumanEvalSimX.

Figures & tables

Panel of Figure 1 — Figure 1: We consider environment interaction in two settings. The scope of our analysis is marked in yellow.

Figure 2: Generative process represented as a structural causal model (arrows indicate causality). We observe an informal specification (description) corresponding to an algorithm in an execution context (environment). We seek an executable specification (test suite) and an implementation that satisfies it (code).

	$\mathbb{O}$	selection criterion		$\mathbb{O}$	selection criterion
MBR-Exec Hard ( Shi et al., 2022 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})$	MBR-Exec Soft ( Shi et al., 2022 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})$
AlphaCode ( Li et al., 2022 )	$\mathbb{A}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})$	FunCoder ( Chen et al., 2024a )	$\mathbb{A}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})$
MaxPass Hard (various works ∗ )	$\mathbb{B}$	$\hat{\mathbb{E}}_{t_{1:m}}[R^{\infty}]$	MaxPass Soft (ours)	$\mathbb{B}$	$\hat{\mathbb{E}}_{t_{1:m}}[R]$
CodeT Hard ( Chen et al., 2023 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})\hat{\mathbb{E}}_{t_{1:m}}[R]$	CodeT Soft (ours)	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})\hat{\mathbb{E}}_{t_{1:m}}[R]$

Table 1: Selection heuristics maximize pass@1 using different estimators of code correctness. The test output

\mathbb{O}

is binary

\mathbb{B}

(pass/fail) or arbitrary

\mathbb{A}

. The asterisk marks that only single-test test suites were considered by Lahiri et al. (2022) ; Rozière et al. (2022) ; Le et al. (2022) ; Chen et al. (2024b) .

	BigCodeBenchHard (BigCode)			QiskitHumanEvalSim (Qiskit)			LeetCodeDataset (LeetCode)
	Qwen3	GPT-OSS	MM-M2.1	Qwen3	GPT-OSS	MM-M2.1	Qwen3	GPT-OSS	MM-M2.1
Random Selection	$27.43^{\scriptscriptstyle\pm 0.31}$	$37.16^{\scriptscriptstyle\pm 0.63}$	$34.46^{\scriptscriptstyle\pm 0.45}$	$52.73^{\scriptscriptstyle\pm 0.19}$	$41.96^{\scriptscriptstyle\pm 0.36}$	$43.78^{\scriptscriptstyle\pm 0.47}$	$18.60^{\scriptscriptstyle\pm 0.22}$	$69.12^{\scriptscriptstyle\pm 0.31}$	$62.54^{\scriptscriptstyle\pm 0.58}$
MBR-Exec Hard	$29.32^{\scriptscriptstyle\pm 0.43}$	$37.30^{\scriptscriptstyle\pm 0.42}$	$35.95^{\scriptscriptstyle\pm 0.55}$	$54.97^{\scriptscriptstyle\pm 0.13}$	$\mathbf{47.69}^{\scriptscriptstyle\pm 0.40}$	$45.73^{\scriptscriptstyle\pm 0.34}$	$18.33^{\scriptscriptstyle\pm 0.18}$	$\mathbf{72.81}^{\scriptscriptstyle\pm 0.41}$	$73.07^{\scriptscriptstyle\pm 0.33}$
MBR-Exec Soft	$\mathbf{29.73}^{\scriptscriptstyle\pm 0.42}$	$37.30^{\scriptscriptstyle\pm 0.54}$	$\mathbf{36.76}^{\scriptscriptstyle\pm 0.31}$	$54.97^{\scriptscriptstyle\pm 0.16}$	$47.13^{\scriptscriptstyle\pm 0.35}$	$\mathbf{46.71}^{\scriptscriptstyle\pm 0.23}$	$\mathbf{18.60}^{\scriptscriptstyle\pm 0.16}$	$72.19^{\scriptscriptstyle\pm 0.32}$	$\mathbf{73.33}^{\scriptscriptstyle\pm 0.45}$
CodeT Hard	$31.22^{\scriptscriptstyle\pm 0.32}$	$37.30^{\scriptscriptstyle\pm 0.24}$	$36.08^{\scriptscriptstyle\pm 0.59}$	$59.02^{\scriptscriptstyle\pm 0.19}$	$51.33^{\scriptscriptstyle\pm 0.38}$	$50.49^{\scriptscriptstyle\pm 0.23}$	$21.23^{\scriptscriptstyle\pm 0.12}$	$73.25^{\scriptscriptstyle\pm 0.36}$	$78.60^{\scriptscriptstyle\pm 0.24}$
CodeT Soft (ours)	$\mathbf{35.00}^{\scriptscriptstyle\pm 0.36}$	$\mathbf{38.24}^{\scriptscriptstyle\pm 0.34}$	$\mathbf{42.03}^{\scriptscriptstyle\pm 0.42}$	$\mathbf{62.24}^{\scriptscriptstyle\pm 0.28}$	$\mathbf{54.13}^{\scriptscriptstyle\pm 0.27}$	$\mathbf{53.85}^{\scriptscriptstyle\pm 0.33}$	$\mathbf{24.30}^{\scriptscriptstyle\pm 0.23}$	$\mathbf{74.56}^{\scriptscriptstyle\pm 0.49}$	$\underline{\mathbf{79.56}}^{\scriptscriptstyle\pm 0.24}$

Table 2: Pass@1 (%) of post-generation selection heuristics (see Table 1 ) after 10 rounds of independent code and test generation (

m=n=10

). Within a family of heuristics, we bold the more accurate one. The soft variants generally outperform the hard ones, corroborating the insights of Theorem 3.8 , Theorem 3.9 , and Theorem 3.10 . We report mean (

\pm

standard error) over 5 seeds.

Panel of Figure 3 — Figure 3: Pass@1 (%) during $10$ rounds of in-context Thompson sampling using Qwen3-235B-A22B-Instruct-2507. We report similar trends across other members of the Qwen3 family, including hybrids, in Appendix C.3 . W.r.t. Theorem 4.1 , the black solid line denotes the true reward $r$ , and the cyan dashed line $r_{obs}$ . The purple dotted line is the simplified setting of known $r_{hid}$ , which, according to Corollary 4.2 , permits global convergence to $x^{*}$ . We report mean ( $\pm$ standard error) over 5 seeds.

Figure 4: Pass@1 (%) during

10

rounds of backprompting on QiskitHumanEvalSimX. We use a Qwen3-235B-A22B-Instruct-2507. Results are reported as mean (

\pm

standard error) over 5 seeds.

Method	BigCode	Qiskit
Oracle Estimator	$30.68^{\scriptscriptstyle\pm 1.69}$	$16.08^{\scriptscriptstyle\pm 0.54}$
Online Estimator	$27.07^{\scriptscriptstyle\pm 0.79}$	$11.66^{\scriptscriptstyle\pm 1.02}$

Table 3: Online estimation of the irreducible regret

\Delta

(%) on Qwen3-235B-A22B-Instruct-2507 trajectories. We report mean (

\pm

standard error) over 5 seeds.

Figure 5: Solve rate (%) of MiniMax M2.7 during 250 environment interactions on SWE-bench Verified. The reproduction setting applies oracle test patches, and thus matches

\Delta=0

in Corollary 4.2 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure S1 : Pass@1 (%) improvements of common post-generation selection heuristics (see Table 1 and Table 2 ) across BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset. Qwen3-235B-A22B-Instruct-2507, GPT-OSS-120B, or MiniMax-M2.1 are used to independently generate code and tests across multiple rounds. Results are reported as mean ( $\pm$ standard error) over 5 seeds.

Figure S2 : Pass@1 of in-context Thompson sampling via backprompting of self-test feedback using Qwen3-235B-A22B-Instruct-2507 on QiskitHumanEvalSim.

Panel of Figure 11 — Figure S3 : Pass@1 (%) of approximate in-context Thompson sampling via backprompting with Qwen3-235B-A22B-Instruct-2507, GPT-OSS-120B, and MiniMax-M2.1 across BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset. In the language of Theorem 4.1 , the black solid line corresponds to the true reward $r$ , and the cyan dashed line to $r_{obs}$ . The purple dotted line corresponds to the case of known $r_{hid}$ , which according to Corollary 4.2 permits global convergence to $x^{*}$ . These experiments were conducted with insight reformulation and independent factorization ( $p(c,t)=p(c)\cdot p(t)$ ). Results are reported as mean ( $\pm$ standard error) over 5 random seeds.

	BigCode	Qiskit	LeetCode
Qwen3-235B-A22B	33.38 -> 33.11	53.0 0 -> 56.22	24.65 -> 72.11
Qwen3-Next	35.95 -> 32.7	50.35 -> 56.08	28.07 -> 67.28
Qwen3-Coder-Next	36.08 -> 30.27	43.5 0 -> 52.31	39.3 0 -> 64.39
Qwen3.5-397B-A17B	37.97 -> 37.3	55.66 -> 59.3	66.96 -> 73.98

Appendix

Table S1 : Initial and final pass@1 (%) of members of the Qwen3(.5) model family during 10 rounds of backprompting self-test feedback. We report the mean across 5 seeds. For detailed trajectories, see Figure S4 .

Panel of Figure 13 — Figure S4 : Pass@1 (%) of backprompting concatenated summaries of self-test feedback using the Qwen3 model family. Results are reported as mean ( $\pm$ standard error) over 5 seeds.

Compression	Factorization	BigCodeBenchHard	QiskitHumanEvalSim	LeetCodeDataset
Insights Reformulation	$p(c)\cdot p(t)$	$30.81^{\pm 0.28}$ ( $-2.16^{\pm 0.32}$ )	$54.55^{\pm 0.33}$ ( $2.24^{\pm 0.33}$ )	$61.05^{\pm 0.32}$ ( $34.56^{\pm 0.48}$ )
Insights Reformulation	$p(t\mid c)\cdot p(c)$	$31.89^{\pm 0.45}$ ( $-0.68^{\pm 0.51}$ )	$54.27^{\pm 0.39}$ ( $2.94^{\pm 0.90}$ )	$58.60^{\pm 0.47}$ ( $34.65^{\pm 0.72}$ )
Insights Reformulation	$p(c\mid t)\cdot p(t)$	$\textbf{34.05}^{\pm 0.31}$ ( $-0.81^{\pm 0.46}$ )	$54.97^{\pm 0.19}$ ( $3.08^{\pm 0.40}$ )	$60.61^{\pm 0.48}$ ( $36.40^{\pm 0.30}$ )
Summary Concatenation	$p(c)\cdot p(t)$	$32.84^{\pm 0.45}$ ( $-0.27^{\pm 0.49}$ )	$55.52^{\pm 0.36}$ ( $3.92^{\pm 0.46}$ )	$\underline{71.14}^{\pm 0.24}$ ( $42.02^{\pm 0.32}$ )
Summary Concatenation	$p(t\mid c)\cdot p(c)$	$32.03^{\pm 0.44}$ ( $-0.41^{\pm 0.58}$ )	$\underline{56.08}^{\pm 0.55}$ ( $5.45^{\pm 0.92}$ )	$68.42^{\pm 0.31}$ ( $44.74^{\pm 1.00}$ )
Summary Concatenation	$p(c\mid t)\cdot p(t)$	$\underline{33.11}^{\pm 0.51}$ ( $-0.27^{\pm 0.75}$ )	$\textbf{56.22}^{\pm 0.45}$ ( $3.22^{\pm 0.70}$ )	$\textbf{72.11}^{\pm 0.08}$ ( $47.46^{\pm 0.39}$ )

Appendix

Table S2 : Pass@1 (%) after 10 rounds of in-context Thompson sampling (with improvements during generation in parenthesis) for different factorization of

p(c,t)

and different test report compression methods. Results reported as mean (

\pm

standard error) over 5 random seeds using Qwen3-235B-A22B-Instruct. For each column, the best configuration is emphasized and the second best is underlined.

Selection	Backprompting	BigCode	Qiskit	LeetCode
Random	No	$27.43^{\scriptscriptstyle\pm 0.31}$	$52.73^{\scriptscriptstyle\pm 0.19}$	$18.60^{\scriptscriptstyle\pm 0.22}$
Last	Self-Test FB	$\mathbf{33.11}^{\scriptscriptstyle\pm 0.51}$	$\mathbf{56.22}^{\scriptscriptstyle\pm 0.45}$	$\mathbf{72.11}^{\scriptscriptstyle\pm 0.08}$
MP Soft	No	$35.41^{\scriptscriptstyle\pm 0.38}$	$\underline{\mathbf{63.22}}^{\scriptscriptstyle\pm 0.19}$	$25.09^{\scriptscriptstyle\pm 0.29}$
MP Soft	Self-Test FB	$\underline{\mathbf{36.22}}^{\scriptscriptstyle\pm 0.47}$	$60.84^{\scriptscriptstyle\pm 0.36}$	$\underline{\mathbf{73.42}}^{\scriptscriptstyle\pm 0.17}$

Appendix

Table S3 : Pass@1 (%) comparison of backprompting and post-gen selection. Each solution produced 10 code-test pairs. We use a Qwen3-235B-A22B-Instruct-2507 and report mean (

\pm

standard error) over 5 seeds.

Panel of Figure 16 — Figure S5 : Distribution of average test failure rate (computed across all generated programs) for $261,566$ test cases generated by Qwen3-235B-A22B-Instruct on BigCode, Qiskit, and LeetCode.

Dataset	Round	Average Test Suite Length	Precision
BigCodeBenchHard	1	9.86	41.4%
BigCodeBenchHard	5	10.50	41.9%
BigCodeBenchHard	10	10.73	37.8%
QiskitHumanEvalSim	1	7.17	73.1%
QiskitHumanEvalSim	5	7.69	74.2%
QiskitHumanEvalSim	10	7.80	71.2%

Appendix

Table S4 : Average test suite length and precision (positive predictive value) across 10 rounds of in-context Thompson sampling using the Qwen3-235B-A22B-Instruct model.

Round	BigCodeBenchHard	QiskitHumanEvalSim	LeetCodeDataset
4	56.40%	77.48%	75.29%
5	62.12%	80.70%	81.37%
6	66.08%	82.84%	83.13%
7	68.29%	84.38%	85.56%
8	71.71%	87.65%	87.02%
9	73.65%	87.69%	88.36%

Appendix

Table S5 : Collision probability per round of in-context Thompson sampling using Qwen3-235B-A22B-Instruct (averaged across a sliding window of size 3 to account for non-stationarity).

Dataset	Strategy	# Test Suites	Spearman’s $\rho$	Kendall’s $\tau$	SNR (bootstrapped)
BigCodeBenchHard	MaxPass Hard	5	0.500	0.494	3.01
	MaxPass Soft	5	0.495	0.481	151.40
LeetCodeDataset	MaxPass Hard	5	0.864	0.861	5.11
	MaxPass Soft	5	0.804	0.793	553.83
QiskitHumanEvalSim	MaxPass Hard	5	0.787	0.783	2.79
	MaxPass Soft	5	0.692	0.680	53.53

Appendix

Table S6 : Rank correlation and SNR analysis at 5 and 10 rounds of independent generation using Qwen3-235B-A22B-Instruct. This is the primary setting investigated for post-generation selection.

Dataset	B4	MaxPass Soft
BigCodeBenchHard	33.24% $\pm$ 0.48%	35.41% $\pm$ 0.75%
LeetCodeDataset	25.18% $\pm$ 0.42%	25.09% $\pm$ 0.57%
QiskitHumanEvalSim	60.56% $\pm$ 0.32%	63.22% $\pm$ 0.38%

Appendix

Table S7 : Pass@1 (%) performance comparing B4 against MaxPass Soft as post-generation selection mechanisms after 10 rounds of independent generation using Qwen3-235B-A22B-Instruct.

Execution-Guided Code Generation

Execution-guided code generation uses program runs, tests, and tool feedback to improve generated code and assess its correctness.

See all 67 papers

TDD-Agent: Test-Driven Reasoning for Code Generation
Aug 17, 2026Hongyue Yu, Kefan Li, Jiakun Li +4Code Generation Execution-Guided Code Generation
National College for Excellent Engineers, Beihang University · School of Computer Science and Engineering, Beihang University · Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University +1
Planning to Explore: Curiosity-Driven Planning for LLM Test Generation
Apr 6, 2026Alfonso Amayuelas, Firas Laakom, Piotr Piękos +5Automated Test Generation LLM Planning
University of California, Santa Barbara · King Abdullah University of Science and Technology
Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation
Sep 18, 2026Ana Nunez, Peyman NajafiradRL for Code Generation Code Generation
Secure AI and Autonomy Lab University of Texas at San Antonio

Aug 17, 2026J/K move · Enter open · S save

TDD-Agent: Test-Driven Reasoning for Code Generation

Hongyue Yu, Kefan Li, Jiakun Li, Hongzheng Chai+3

National College for Excellent Engineers, Beihang University · School of Computer Science and Engineering, Beihang University · Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University+1

Code Generation Execution-Guided Code Generation Repository-Level Code Generation Automated Test Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect. In this paper, we introduce TDD-Agent, which operationalizes the test-driven development paradigm for code generation. TDD-Agent first prompts the model to generate executable tests, encouraging it to clarify expected behaviors before implementation, and then performs iterative dual-track refinement over both the generated code and tests using execution feedback. We first isolate the effect of test-first reasoning through a prompt variant TDD-prompt on LiveCodeBench, where it consistently improves upon reasoning-based prompting baselines. Building on this finding, we evaluate the full TDD-Agent framework on RepoEval, a repository-level benchmark, and show that it consistently outperforms retrieval-based and agent-based baselines. Additional analyses show that iterative refinement improves not only code correctness but also the effectiveness of the generated tests, yielding higher pass rates, coverage, and mutation scores, suggesting that tests can serve as evolving reasoning artifacts rather than fixed validators. Our source code is available at https://anonymous.4open.science/r/TDD-Agent-Framework-6370/.

Abstract

Execution-Guided Code Generation

Explore similar work

TDD-Agent: Test-Driven Reasoning for Code Generation

Planning to Explore: Curiosity-Driven Planning for LLM Test Generation

Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation