A Theoretical Analysis of Test-Driven Code Generation

Nicolas Menet; Michael Hersche; Andreas Krause; Abbas Rahimi

cs.SEFeb 5, 2026

A Theoretical Analysis of Test-Driven Code Generation

Authors: Nicolas Menet, Michael Hersche, Andreas Krause, Abbas Rahimi

Abstract

Code assistants are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant paradigms: code selection after generation using the execution environment, and code generation conditioned on environment feedback. First, we formalize several well-established selection heuristics as environment-aware estimators of code correctness. We theoretically prove that estimators based on fuzzy functional similarity add an inductive bias and strictly dominate estimators based on functional equivalence in terms of signal-to-noise ratio. Second, we frame backprompting as an in-context approximation of Thompson sampling. We derive a novel regret bound for reward functions with unobservable components, theoretically explaining why the effectiveness of backprompting is limited by the ambiguity of the informal task description (an irreducible regret). Using seven state-of-the-art open weight models, we corroborate these findings across BigCodeBenchHard, LeetCodeDataset, QiskitHumanEvalSim, and SWE-bench Verified. Our formalization also suggests how to improve task descriptions effectively, leading to an improved benchmark for LLM-assisted quantum programming, QiskitHumanEvalSimX.

Figures & tables

Panel of Figure 1 — Figure 1: We consider environment interaction in two settings. The scope of our analysis is marked in yellow.

Figure 2: Generative process represented as a structural causal model (arrows indicate causality). We observe an informal specification (description) corresponding to an algorithm in an execution context (environment). We seek an executable specification (test suite) and an implementation that satisfies it (code).

	$\mathbb{O}$	selection criterion		$\mathbb{O}$	selection criterion
MBR-Exec Hard ( Shi et al., 2022 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})$	MBR-Exec Soft ( Shi et al., 2022 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})$
AlphaCode ( Li et al., 2022 )	$\mathbb{A}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})$	FunCoder ( Chen et al., 2024a )	$\mathbb{A}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})$
MaxPass Hard (various works ∗ )	$\mathbb{B}$	$\hat{\mathbb{E}}_{t_{1:m}}[R^{\infty}]$	MaxPass Soft (ours)	$\mathbb{B}$	$\hat{\mathbb{E}}_{t_{1:m}}[R]$
CodeT Hard ( Chen et al., 2023 )	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{\infty}_{c})\hat{\mathbb{E}}_{t_{1:m}}[R]$	CodeT Soft (ours)	$\mathbb{B}$	$\hat{p}_{n,m}(\mathcal{N}^{1}_{c})\hat{\mathbb{E}}_{t_{1:m}}[R]$

Table 1: Selection heuristics maximize pass@1 using different estimators of code correctness. The test output

\mathbb{O}

is binary

\mathbb{B}

(pass/fail) or arbitrary

\mathbb{A}

. The asterisk marks that only single-test test suites were considered by Lahiri et al. (2022) ; Rozière et al. (2022) ; Le et al. (2022) ; Chen et al. (2024b) .

	BigCodeBenchHard (BigCode)			QiskitHumanEvalSim (Qiskit)			LeetCodeDataset (LeetCode)
	Qwen3	GPT-OSS	MM-M2.1	Qwen3	GPT-OSS	MM-M2.1	Qwen3	GPT-OSS	MM-M2.1
Random Selection	$27.43^{\scriptscriptstyle\pm 0.31}$	$37.16^{\scriptscriptstyle\pm 0.63}$	$34.46^{\scriptscriptstyle\pm 0.45}$	$52.73^{\scriptscriptstyle\pm 0.19}$	$41.96^{\scriptscriptstyle\pm 0.36}$	$43.78^{\scriptscriptstyle\pm 0.47}$	$18.60^{\scriptscriptstyle\pm 0.22}$	$69.12^{\scriptscriptstyle\pm 0.31}$	$62.54^{\scriptscriptstyle\pm 0.58}$
MBR-Exec Hard	$29.32^{\scriptscriptstyle\pm 0.43}$	$37.30^{\scriptscriptstyle\pm 0.42}$	$35.95^{\scriptscriptstyle\pm 0.55}$	$54.97^{\scriptscriptstyle\pm 0.13}$	$\mathbf{47.69}^{\scriptscriptstyle\pm 0.40}$	$45.73^{\scriptscriptstyle\pm 0.34}$	$18.33^{\scriptscriptstyle\pm 0.18}$	$\mathbf{72.81}^{\scriptscriptstyle\pm 0.41}$	$73.07^{\scriptscriptstyle\pm 0.33}$
MBR-Exec Soft	$\mathbf{29.73}^{\scriptscriptstyle\pm 0.42}$	$37.30^{\scriptscriptstyle\pm 0.54}$	$\mathbf{36.76}^{\scriptscriptstyle\pm 0.31}$	$54.97^{\scriptscriptstyle\pm 0.16}$	$47.13^{\scriptscriptstyle\pm 0.35}$	$\mathbf{46.71}^{\scriptscriptstyle\pm 0.23}$	$\mathbf{18.60}^{\scriptscriptstyle\pm 0.16}$	$72.19^{\scriptscriptstyle\pm 0.32}$	$\mathbf{73.33}^{\scriptscriptstyle\pm 0.45}$
CodeT Hard	$31.22^{\scriptscriptstyle\pm 0.32}$	$37.30^{\scriptscriptstyle\pm 0.24}$	$36.08^{\scriptscriptstyle\pm 0.59}$	$59.02^{\scriptscriptstyle\pm 0.19}$	$51.33^{\scriptscriptstyle\pm 0.38}$	$50.49^{\scriptscriptstyle\pm 0.23}$	$21.23^{\scriptscriptstyle\pm 0.12}$	$73.25^{\scriptscriptstyle\pm 0.36}$	$78.60^{\scriptscriptstyle\pm 0.24}$
CodeT Soft (ours)	$\mathbf{35.00}^{\scriptscriptstyle\pm 0.36}$	$\mathbf{38.24}^{\scriptscriptstyle\pm 0.34}$	$\mathbf{42.03}^{\scriptscriptstyle\pm 0.42}$	$\mathbf{62.24}^{\scriptscriptstyle\pm 0.28}$	$\mathbf{54.13}^{\scriptscriptstyle\pm 0.27}$	$\mathbf{53.85}^{\scriptscriptstyle\pm 0.33}$	$\mathbf{24.30}^{\scriptscriptstyle\pm 0.23}$	$\mathbf{74.56}^{\scriptscriptstyle\pm 0.49}$	$\underline{\mathbf{79.56}}^{\scriptscriptstyle\pm 0.24}$

Table 2: Pass@1 (%) of post-generation selection heuristics (see Table 1 ) after 10 rounds of independent code and test generation (

m=n=10

). Within a family of heuristics, we bold the more accurate one. The soft variants generally outperform the hard ones, corroborating the insights of Theorem 3.8 , Theorem 3.9 , and Theorem 3.10 . We report mean (

\pm

standard error) over 5 seeds.

Panel of Figure 3 — Figure 3: Pass@1 (%) during $10$ rounds of in-context Thompson sampling using Qwen3-235B-A22B-Instruct-2507. We report similar trends across other members of the Qwen3 family, including hybrids, in Appendix C.3 . W.r.t. Theorem 4.1 , the black solid line denotes the true reward $r$ , and the cyan dashed line $r_{obs}$ . The purple dotted line is the simplified setting of known $r_{hid}$ , which, according to Corollary 4.2 , permits global convergence to $x^{*}$ . We report mean ( $\pm$ standard error) over 5 seeds.

Figure 4: Pass@1 (%) during

10

rounds of backprompting on QiskitHumanEvalSimX. We use a Qwen3-235B-A22B-Instruct-2507. Results are reported as mean (

\pm

standard error) over 5 seeds.

Method	BigCode	Qiskit
Oracle Estimator	$30.68^{\scriptscriptstyle\pm 1.69}$	$16.08^{\scriptscriptstyle\pm 0.54}$
Online Estimator	$27.07^{\scriptscriptstyle\pm 0.79}$	$11.66^{\scriptscriptstyle\pm 1.02}$

Table 3: Online estimation of the irreducible regret

\Delta

(%) on Qwen3-235B-A22B-Instruct-2507 trajectories. We report mean (

\pm

standard error) over 5 seeds.

Figure 5: Solve rate (%) of MiniMax M2.7 during 250 environment interactions on SWE-bench Verified. The reproduction setting applies oracle test patches, and thus matches

\Delta=0

in Corollary 4.2 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure S1 : Pass@1 (%) improvements of common post-generation selection heuristics (see Table 1 and Table 2 ) across BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset. Qwen3-235B-A22B-Instruct-2507, GPT-OSS-120B, or MiniMax-M2.1 are used to independently generate code and tests across multiple rounds. Results are reported as mean ( $\pm$ standard error) over 5 seeds.

Figure S2 : Pass@1 of in-context Thompson sampling via backprompting of self-test feedback using Qwen3-235B-A22B-Instruct-2507 on QiskitHumanEvalSim.

Panel of Figure 11 — Figure S3 : Pass@1 (%) of approximate in-context Thompson sampling via backprompting with Qwen3-235B-A22B-Instruct-2507, GPT-OSS-120B, and MiniMax-M2.1 across BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset. In the language of Theorem 4.1 , the black solid line corresponds to the true reward $r$ , and the cyan dashed line to $r_{obs}$ . The purple dotted line corresponds to the case of known $r_{hid}$ , which according to Corollary 4.2 permits global convergence to $x^{*}$ . These experiments were conducted with insight reformulation and independent factorization ( $p(c,t)=p(c)\cdot p(t)$ ). Results are reported as mean ( $\pm$ standard error) over 5 random seeds.

	BigCode	Qiskit	LeetCode
Qwen3-235B-A22B	33.38 -> 33.11	53.0 0 -> 56.22	24.65 -> 72.11
Qwen3-Next	35.95 -> 32.7	50.35 -> 56.08	28.07 -> 67.28
Qwen3-Coder-Next	36.08 -> 30.27	43.5 0 -> 52.31	39.3 0 -> 64.39
Qwen3.5-397B-A17B	37.97 -> 37.3	55.66 -> 59.3	66.96 -> 73.98

Appendix

Table S1 : Initial and final pass@1 (%) of members of the Qwen3(.5) model family during 10 rounds of backprompting self-test feedback. We report the mean across 5 seeds. For detailed trajectories, see Figure S4 .

Panel of Figure 13 — Figure S4 : Pass@1 (%) of backprompting concatenated summaries of self-test feedback using the Qwen3 model family. Results are reported as mean ( $\pm$ standard error) over 5 seeds.

Compression	Factorization	BigCodeBenchHard	QiskitHumanEvalSim	LeetCodeDataset
Insights Reformulation	$p(c)\cdot p(t)$	$30.81^{\pm 0.28}$ ( $-2.16^{\pm 0.32}$ )	$54.55^{\pm 0.33}$ ( $2.24^{\pm 0.33}$ )	$61.05^{\pm 0.32}$ ( $34.56^{\pm 0.48}$ )
Insights Reformulation	$p(t\mid c)\cdot p(c)$	$31.89^{\pm 0.45}$ ( $-0.68^{\pm 0.51}$ )	$54.27^{\pm 0.39}$ ( $2.94^{\pm 0.90}$ )	$58.60^{\pm 0.47}$ ( $34.65^{\pm 0.72}$ )
Insights Reformulation	$p(c\mid t)\cdot p(t)$	$\textbf{34.05}^{\pm 0.31}$ ( $-0.81^{\pm 0.46}$ )	$54.97^{\pm 0.19}$ ( $3.08^{\pm 0.40}$ )	$60.61^{\pm 0.48}$ ( $36.40^{\pm 0.30}$ )
Summary Concatenation	$p(c)\cdot p(t)$	$32.84^{\pm 0.45}$ ( $-0.27^{\pm 0.49}$ )	$55.52^{\pm 0.36}$ ( $3.92^{\pm 0.46}$ )	$\underline{71.14}^{\pm 0.24}$ ( $42.02^{\pm 0.32}$ )
Summary Concatenation	$p(t\mid c)\cdot p(c)$	$32.03^{\pm 0.44}$ ( $-0.41^{\pm 0.58}$ )	$\underline{56.08}^{\pm 0.55}$ ( $5.45^{\pm 0.92}$ )	$68.42^{\pm 0.31}$ ( $44.74^{\pm 1.00}$ )
Summary Concatenation	$p(c\mid t)\cdot p(t)$	$\underline{33.11}^{\pm 0.51}$ ( $-0.27^{\pm 0.75}$ )	$\textbf{56.22}^{\pm 0.45}$ ( $3.22^{\pm 0.70}$ )	$\textbf{72.11}^{\pm 0.08}$ ( $47.46^{\pm 0.39}$ )

Appendix

Table S2 : Pass@1 (%) after 10 rounds of in-context Thompson sampling (with improvements during generation in parenthesis) for different factorization of

p(c,t)

and different test report compression methods. Results reported as mean (

\pm

standard error) over 5 random seeds using Qwen3-235B-A22B-Instruct. For each column, the best configuration is emphasized and the second best is underlined.

Selection	Backprompting	BigCode	Qiskit	LeetCode
Random	No	$27.43^{\scriptscriptstyle\pm 0.31}$	$52.73^{\scriptscriptstyle\pm 0.19}$	$18.60^{\scriptscriptstyle\pm 0.22}$
Last	Self-Test FB	$\mathbf{33.11}^{\scriptscriptstyle\pm 0.51}$	$\mathbf{56.22}^{\scriptscriptstyle\pm 0.45}$	$\mathbf{72.11}^{\scriptscriptstyle\pm 0.08}$
MP Soft	No	$35.41^{\scriptscriptstyle\pm 0.38}$	$\underline{\mathbf{63.22}}^{\scriptscriptstyle\pm 0.19}$	$25.09^{\scriptscriptstyle\pm 0.29}$
MP Soft	Self-Test FB	$\underline{\mathbf{36.22}}^{\scriptscriptstyle\pm 0.47}$	$60.84^{\scriptscriptstyle\pm 0.36}$	$\underline{\mathbf{73.42}}^{\scriptscriptstyle\pm 0.17}$

Appendix

Table S3 : Pass@1 (%) comparison of backprompting and post-gen selection. Each solution produced 10 code-test pairs. We use a Qwen3-235B-A22B-Instruct-2507 and report mean (

\pm

standard error) over 5 seeds.

Panel of Figure 16 — Figure S5 : Distribution of average test failure rate (computed across all generated programs) for $261,566$ test cases generated by Qwen3-235B-A22B-Instruct on BigCode, Qiskit, and LeetCode.

Dataset	Round	Average Test Suite Length	Precision
BigCodeBenchHard	1	9.86	41.4%
BigCodeBenchHard	5	10.50	41.9%
BigCodeBenchHard	10	10.73	37.8%
QiskitHumanEvalSim	1	7.17	73.1%
QiskitHumanEvalSim	5	7.69	74.2%
QiskitHumanEvalSim	10	7.80	71.2%

Appendix

Table S4 : Average test suite length and precision (positive predictive value) across 10 rounds of in-context Thompson sampling using the Qwen3-235B-A22B-Instruct model.

Round	BigCodeBenchHard	QiskitHumanEvalSim	LeetCodeDataset
4	56.40%	77.48%	75.29%
5	62.12%	80.70%	81.37%
6	66.08%	82.84%	83.13%
7	68.29%	84.38%	85.56%
8	71.71%	87.65%	87.02%
9	73.65%	87.69%	88.36%

Appendix

Table S5 : Collision probability per round of in-context Thompson sampling using Qwen3-235B-A22B-Instruct (averaged across a sliding window of size 3 to account for non-stationarity).

Dataset	Strategy	# Test Suites	Spearman’s $\rho$	Kendall’s $\tau$	SNR (bootstrapped)
BigCodeBenchHard	MaxPass Hard	5	0.500	0.494	3.01
	MaxPass Soft	5	0.495	0.481	151.40
LeetCodeDataset	MaxPass Hard	5	0.864	0.861	5.11
	MaxPass Soft	5	0.804	0.793	553.83
QiskitHumanEvalSim	MaxPass Hard	5	0.787	0.783	2.79
	MaxPass Soft	5	0.692	0.680	53.53

Appendix

Table S6 : Rank correlation and SNR analysis at 5 and 10 rounds of independent generation using Qwen3-235B-A22B-Instruct. This is the primary setting investigated for post-generation selection.

Dataset	B4	MaxPass Soft
BigCodeBenchHard	33.24% $\pm$ 0.48%	35.41% $\pm$ 0.75%
LeetCodeDataset	25.18% $\pm$ 0.42%	25.09% $\pm$ 0.57%
QiskitHumanEvalSim	60.56% $\pm$ 0.32%	63.22% $\pm$ 0.38%

Appendix

Table S7 : Pass@1 (%) performance comparing B4 against MaxPass Soft as post-generation selection mechanisms after 10 rounds of independent generation using Qwen3-235B-A22B-Instruct.

Execution-Guided Code Generation

Execution-guided code generation uses program runs, tests, and tool feedback to improve generated code and assess its correctness.

See all 67 papers

Aug 17, 2026

TDD-Agent: Test-Driven Reasoning for Code Generation

Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect. In this paper, we introduce TDD-Agent, which operationalizes the test-driven development paradigm for code generation. TDD-Agent first prompts the model to generate executable tests, encouraging it to clarify expected behaviors before implementation, and then performs iterative dual-track refinement over both the generated code and tests using execution feedback. We first isolate the effect of test-first reasoning through a prompt variant TDD-prompt on LiveCodeBench, where it consistently improves upon reasoning-based prompting baselines. Building on this finding, we evaluate the full TDD-Agent framework on RepoEval, a repository-level benchmark, and show that it consistently outperforms retrieval-based and agent-based baselines. Additional analyses show that iterative refinement improves not only code correctness but also the effectiveness of the generated tests, yielding higher pass rates, coverage, and mutation scores, suggesting that tests can serve as evolving reasoning artifacts rather than fixed validators. Our source code is available at https://anonymous.4open.science/r/TDD-Agent-Framework-6370/.

Hongyue Yu, Kefan Li, Jiakun Li +4

National College for Excellent Engineers, Beihang University · School of Computer Science and Engineering, Beihang University · Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University +1

Code Generation Execution-Guided Code Generation Repository-Level Code Generation Automated Test Generation

Ask PDF

Apr 6, 2026

Planning to Explore: Curiosity-Driven Planning for LLM Test Generation

The use of LLMs for code generation has naturally extended to code testing and evaluation. As codebases grow in size and complexity, so does the need for automated test generation. Current approaches for LLM-based test generation rely on strategies that maximize immediate coverage gain, a greedy approach that plateaus on code where reaching deep branches requires setup steps that individually yield zero new coverage. Drawing on principles of Bayesian exploration, we treat the program's branch structure as an unknown environment, and an evolving coverage map as a proxy probabilistic posterior representing what the LLM has discovered so far. Our method, CovQValue, feeds the coverage map back to the LLM, generates diverse candidate plans in parallel, and selects the most informative plan by LLM-estimated Q-values, seeking actions that balance immediate branch discovery with future reachability. Our method outperforms greedy selection on TestGenEval Lite, achieving 51-77% higher branch coverage across three popular LLMs and winning on 77-84% of targets. In addition, we build a benchmark for iterative test generation, RepoExploreBench, where they achieve 40-74%. These results show the potential of curiosity-driven planning methods for LLM-based exploration, enabling more effective discovery of program behavior through sequential interaction

Alfonso Amayuelas, Firas Laakom, Piotr Piękos +5

University of California, Santa Barbara · King Abdullah University of Science and Technology

Automated Test Generation LLM Planning Exploration-Exploitation Tradeoff

Ask PDF

Sep 18, 2026

Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation

Self-play methods that co-train a single language model as both coder and test author promise to move code-generation RL beyond fixed test suites, but they suffer from two coupled pathologies: permissiveness collapse, where pass-rate rewards are maximised by trivial, non-discriminative tests, and concentration bias, where i.i.d. sampled tests cluster on modal inputs and inflate estimator variance. We introduce CoVer (Co-trained Coder and Verifier), a single-policy GRPO framework that addresses both failure modes. First, an information-gain (IG) reward scores each self-generated test by the mutual information between its pass/fail vector and a graded, ground-truth-anchored correctness signal y [0, 1] m, gated by the sign of their covariance so that only positively discriminative tests receive reward. Second, a three-stage diversity-aware selection step prunes a candidate pool to a behaviourally non-redundant suite (invalidity, input-string, execution-profile filtering), raising the effective sample size of the IG estimator at fixed execution budget. On five benchmarks (LiveBench, MBPP, LiveCodeBench, CodeContests, Code-Forces), CoVer raises one-shot pass@1 by +5.8 points at 7B and +7.1 points at 14B over the Qwen2.5-Instruct backbone, and achieves the highest macro-average among all compared methods at both scales. As a drop-in backbone inside the CodeT ranking pipeline, CoVer-7B adds +3.5 points, demonstrating the dual benefit of co-training for both generation and selection.

Ana Nunez, Peyman Najafirad

Secure AI and Autonomy Lab University of Texas at San Antonio

RL for Code Generation Code Generation Reinforcement Learning with Verifiable Rewards Automated Test Generation

Ask PDF

Abstract

Execution-Guided Code Generation

Explore similar work

TDD-Agent: Test-Driven Reasoning for Code Generation

Planning to Explore: Curiosity-Driven Planning for LLM Test Generation

Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation