The use of LLMs for code generation has naturally extended to code testing and evaluation. As codebases grow in size and complexity, so does the need for automated test generation. Current approaches for LLM-based test generation rely on strategies that maximize immediate coverage gain, a greedy approach that plateaus on code where reaching deep branches requires setup steps that individually yield zero new coverage. Drawing on principles of Bayesian exploration, we treat the program's branch structure as an unknown environment, and an evolving coverage map as a proxy probabilistic posterior representing what the LLM has discovered so far. Our method, CovQValue, feeds the coverage map back to the LLM, generates diverse candidate plans in parallel, and selects the most informative plan by LLM-estimated Q-values, seeking actions that balance immediate branch discovery with future reachability. Our method outperforms greedy selection on TestGenEval Lite, achieving 51-77% higher branch coverage across three popular LLMs and winning on 77-84% of targets. In addition, we build a benchmark for iterative test generation, RepoExploreBench, where they achieve 40-74%. These results show the potential of curiosity-driven planning methods for LLM-based exploration, enabling more effective discovery of program behavior through sequential interaction
Figures & tables
Figure 1
Definitions
Symbol
Test Generation
Unknown environment
Θ
Program’s branch reachability structure
Prior
p(Θ)
No branches observed
History
h=(a1,o1)⋯(at,ot)
Sequence of (test, coverage result) pairs
Exploration state
p(Θ∣h)
Coverage map Ct (sufficient statistic after t executions)
Action
a
Test plan (sequence of S scripts)
Observation
o
Branches hit by executing the plan
Table 1: Formal mapping from Sun et al. (2011) optimal Bayesian exploration framework to LLM-based test generation.
Figure 1: Overview of CovQvalue . At each round, the LLM generates K diverse candidate plans conditioned on the coverage map. Each plan is scored by its estimated Q-value, information gain plus discounted future reachability (Equation 2 ), and the highest scoring plan is executed. The resulting branch coverage updates the coverage map for next round.
RepoExploreBench
TestGenEval Lite
Gemini 3
GPT-5.4
Mistral
Gemini 3
GPT-5.4
Mistral
Flash
mini
Large 3
Flash
mini
Large 3
Random
43.2 ± 4.5
31.2 ± 3.0
41.2 ± 4.2
64.4 ± 6.3
50.4 ± 4.6
65.9 ± 5.9
Greedy
41.8 ± 3.9
28.6 ± 2.8
42.3 ± 4.4
61.5 ± 6.0
53.2 ± 4.8
65.0 ± 5.8
CovGreedy
51.7 ± 5.7
36.5 ± 3.5
40.0 ± 4.5
77.6 ± 7.3
66.0 ± 6.2
74.7 ± 7.1
CovQValue
72.4 ± 6.5
49.8 ± 5.0
59.1 ± 6.0
108.9 ± 9.0
80.5 ± 7.2
100.0 ± 8.4
Table 2: Mean per-module branch coverage ( ± standard error) for all 3 models. CovQValue vs. Greedy comparisons: p<0.0001 (paired t -test). d : Cohen’s d effect size.
Figure 2: Cumulative branch coverage over execution steps, averaged across three models (Gemini 3 Flash, GPT-5.4 Mini, Mistral Large). CovQValue (ours) continues discovering branches while Random and Greedy stall earlier. Shaded region indicates ± 1 SE.
Figure 3: Mean branch coverage by repository on TestGenEval Lite, averaged across three models. CovQValue outperforms other methods on every repository, with the largest gains over Greedy on sympy (+55%), matplotlib (+52%), and astropy (+55%).
Figure 4: Trade-off between test pass rate and branch coverage. Large dots represent the mean across models, while individual models and benchmark results are represent with transparent dots. Coverage-map strategies discover more branches by generating larger tests at a small cost in pass rate.
Figure 5: Ablation studies on RepoExploreBench (93 targets, Gemini 3 Flash). (a) CovQValue scales steadily with budget. (b) Longer plans yield higher coverage, confirming the benefit of multi-step lookahead. (c) Top: discount factor sensitivity (default in bold). Bottom: component contributions.
Figure 6: Corridor navigation on four RepoExploreBench modules (Gemini 3 Flash). It showcases how Random or Greedy remain flat while CovQValue traverses setup corridors to reach deep branches as the number of steps increase.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Package
Domain
Mod.
Lines
Corridor Structure
click
CLI framework
10
8,293
Command setup → option parsing → type coercion
flask
Web framework
10
5,369
App creation → config → routing → request context
httpx
Async HTTP client
10
6,925
Client config → auth → transport → URL parsing
jinja2
Template engine
12
12,751
Env setup → lexer → parser → AST → compiler
pydantic
Data validation
10
11,727
Model definition → schema → validation → serialization
requests
HTTP client
6
4,375
Session setup → auth → adapters → connection pool
Appendix
Table 3: RepoExploreBench composition. Packages selected from top 150 PyPI downloads (as of March 2026). Corridor structure describes the sequential dependencies that gate access to deep branches.
Gemini 3 Flash
GPT-5.4 Mini
Mistral Large
Strategy
Lines
Pass%
Lines
Pass%
Lines
Pass%
RepoExploreBench
Random
146
84%
113
95%
142
79%
Greedy
142
83%
109
95%
144
80%
CovGreedy
157
57%
123
86%
129
48%
CovQValue
197
71%
151
87%
171
67%
Appendix
Table 4: Line coverage and pass rate. Results consistent with branch coverage.
Figure 7: Mean branch coverage by repository on RepoExploreBench, averaged for all models. CovQValue outperforms all baselines on every repository.
Gemini Flash
GPT-5.4 Mini
Mistral Large 3
Repo
Rnd
Gdy
CG
CQ
Rnd
Gdy
CG
CQ
Rnd
Gdy
CG
CQ
astropy
37
23
49
80
13
18
19
51
35
35
47
76
django
69
71
83
114
61
65
73
90
81
74
76
109
matplotlib
93
93
101
133
53
68
81
103
86
89
106
119
seaborn
10
20
10
10
9
9
10
10
8
8
9
9
flask
10
18
23
23
19
17
21
25
15
18
8
21
Appendix
Table 5: Per-repository branch coverage on TestGenEval Lite by model. Each cell averages across modules in that repo. Mean row shows the per-module average. Best per group in bold.
Gemini Flash
GPT-5.4 Mini
Mistral Large 3
Repo
Rnd
Gdy
CG
CQ
Rnd
Gdy
CG
CQ
Rnd
Gdy
CG
CQ
click
38
28
32
68
23
24
27
29
29
27
26
51
flask
19
24
31
53
28
18
28
39
20
20
17
37
httpx
37
38
45
67
33
33
39
54
35
33
44
54
jinja2
51
46
57
84
29
30
32
56
43
50
40
61
pydantic
70
58
83
88
52
48
58
80
72
66
76
87
Appendix
Table 6: Per-repository branch coverage on RepoExploreBench by model. Each cell averages across modules in that repo. Mean row shows the per-module average Best per group in bold.
Traditional approaches to test case generation often involve manual effort and incur significant computational overhead. Additionally, these approaches are not scalable, and hence, unsuitable for complex software systems. Recently, Large Language Models (LLMs) have been applied to software testing. However, single-shot prompt engineering-based approaches tend to hallucinate and generate redundant test cases, resulting in fewer branches. To handle the above-mentioned limitations, in this paper, we propose FeedbackLLM, a novel automated language-agnostic test case generation framework based on tightly coupled two-stage approach. In the first stage, FeedbackLLM extracts the input constraints by parsing source code and generates the possible test cases. The quality of the test cases is evaluated in the second stage by the following two specialized LLM feedback agents: (i) Line Feedback Agent: extracts the metadata related to missed line executions and (ii) Branch Feedback Agent: extracts the metadata of the unexecuted branch conditions. The above agents operate in a two-stage process, communicating in tandem, and this procedure is repeated for k-steps. Further, we also introduced a redundancy prevention cache to avoid duplicate API requests and avoid unnecessary execution cycles. The performance of the proposed architecture is evaluated on the standard benchmark programs related to C and Python programs. FeedbackLLM demonstrated more line and branch coverage than baseline tools while scaling linearly in execution time.
Recent advances in large language models for test case generation have improved branch coverage via prompt-engineered mutations. However, they still lack principled mechanisms for steering models toward specific high-risk execution branches, limiting their effectiveness for discovering subtle bugs and security vulnerabilities. We propose GLMTest, the first program structure-aware LLM framework for targeted test case generation that seamlessly integrates code property graphs and code semantics using a graph neural network and a language model to condition test case generation on execution branches. This structured conditioning enables controllable and branch-targeted test case generation, thereby potentially enhancing bug and security risk discovery. Experiments on real-world projects show that GLMTest built on a Qwen2.5-Coder-7B-Instruct model improves branch accuracy from 27.4% to 50.2% on TestGenEval benchmark compared with state-of-the-art LLMs, i.e., Claude-Sonnet-4.5 and GPT-4o-mini.
Khang Tran, Khoa Nguyen, Cristian Borcea +1
New Jersey Institute of Technology, Newark, NJ, USA
Large Language Models (LLMs) are increasingly used for automated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit weaknesses such as reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests and how those tests respond to code evolution is therefore essential. We present a large-scale empirical study of LLM-based test generation under program changes. Using an automated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC) across eight LLMs and 22,374 program variants. LLMs achieve strong baseline results, reaching 79% line coverage and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original behavior rather than adaptation to updated semantics. Performance also declines under SPCs despite unchanged functionality: pass rates fall to 79% and branch coverage to 69%. Although SPC edits preserve semantics, they often introduce larger syntactic changes, leading to instability in generated test suites. Models generate more new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our results indicate that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve.
Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar
Virginia Tech, USA · Carnegie Mellon University, USA