cs.SEMar 24, 2026

Evaluating LLM-Based Test Generation Under Software Evolution

Authors: Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar

Organizations: Virginia Tech, USA · Carnegie Mellon University, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly used for automated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit weaknesses such as reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests and how those tests respond to code evolution is therefore essential. We present a large-scale empirical study of LLM-based test generation under program changes. Using an automated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC) across eight LLMs and 22,374 program variants. LLMs achieve strong baseline results, reaching 79% line coverage and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original behavior rather than adaptation to updated semantics. Performance also declines under SPCs despite unchanged functionality: pass rates fall to 79% and branch coverage to 69%. Although SPC edits preserve semantics, they often introduce larger syntactic changes, leading to instability in generated test suites. Models generate more new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our results indicate that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve.

Figures & tables

Figure 2. Methodology Overview
ModificationTypeDescriptionExample
Semantic-Altering Code Changes(SACs)
Boundary ShiftSemantic-AlteringModifies the inclusive or exclusive range of a loop or array index.range(n) →\rightarrow range(n-1)
Changed Boolean LogicSemantic-AlteringSwitches boolean operators to alter decision logic.a && b →\rightarrow a || b
Changed ArithmeticSemantic-AlteringSwitches arithmetic operators to alter calculation values.a + b →\rightarrow a - b
Argument SwapSemantic-AlteringReorders arguments of the same type in a function call, shifting the functional role of the inputs within the program logic.f(sum, price) →\rightarrow f(price, sum)
Variable Role RebindingSemantic-AlteringReassigns a variable’s logical role.for x in data: total += x →\rightarrow for total in data: total += total
Table 1. Types of Semantic Altering and Semantic Preserving Code Changes Applied to Seed Programs
ModelCreatorSizeType
GPT-OSS ( OpenAI, 2025b )OpenAI20 BOpen-source
Nemotron-3-Nano ( NVIDIA, 2025 )NVIDIA30 BOpen-source
GPT-5 ( Singh and others, 2026 )OpenAIUndisclosedClosed-source
GPT-5.2 ( OpenAI, 2025a )OpenAIUndisclosedClosed-source
Claude 4.5 Haiku ( Anthropic, 2025 )AnthropicUndisclosedClosed-source
Claude 4.6 Sonnet ( Anthropic, 2026 )AnthropicUndisclosedClosed-source
Table 2. LLMs Evaluated
Figure 4. Performance degradation of all LLM-generated test suites across Semantic-Altering Code Changes
Figure 5. Programs evaluated per model to achieve exactly 100 fully passing baseline test suites. The horizontal gap highlights the increased difficulty of test generation in Python compared to Java.
Figure 6. Performance degradation of all LLM-generated test suites across Semantic-Preserving Code Changes
Figure 7. Test pass rates and branch coverage across the original baseline, SPCs, and SACs, broken down by evaluated LLM.
Figure 8. Failure analysis of generated tests under SACs
Figure 9. Regression awareness of LLM-generated test suites.
More in this topic

Automated Test Generation

Automated test generation creates inputs, scenarios, and checks to expose software and system defects with less manual effort.

See all 125 papers

Explore similar work

CardsList
Jul 24, 2026

Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.

Junda Zhao, Shurui Zhou, Eldan Cohen

University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada

LLM EvaluationAutomated Software TestingMutation TestingAutomated Test Generation
AskPDF
Jul 24, 2026

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.

Junda Zhao, Shurui Zhou, Eldan Cohen

University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada

LLM PromptingSpecification-Based TestingAutomated Test GenerationGenerative AI Evaluation
AskPDF
May 21, 2026

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

Evaluating software engineering capabilities has become a core component of modern large language models (LLMs); however, the key bottleneck hindering further scaling lies not in the scarcity of high-quality solutions, but in the lack of high-quality test suites. Test suites are indispensable both for synthesizing program repair trajectories and for providing precise feedback signals in reinforcement learning. Unfortunately, due to the high cost and difficulty of annotation, high-quality test suites have long been hard to obtain, while those automatically generated by LLMs tend to be superficial and lack sufficient discriminative power. As a first step toward constructing high-quality test suites, we introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites. The benchmark characterizes test suites by introducing systematically mutated solutions that attempt to ``fool'' the test suites and pass validation. We further propose an agentic, language-agnostic framework for automatically generating complex mutants. Our benchmark consists of 2,636 mutated variants derived from 800 original instances and includes a multilingual subset spanning nine programming languages. Experiments on seven LLMs reveal that even DeepSeek-V3.1 achieves only 10.20% verification and 36.15% detection rates, highlighting the inadequacy of current LLMs. Additionally, our agentic mutation strategy enhances realism, reducing average detection rates from 71.04% to 39.81% compared to conventional methods. These findings expose persistent deficiencies in the ability of current LLMs to generate reliable and discriminative test suites.

Yuxuan Sun, Yuze Zhao, Yufeng Wang +6

1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3

LLM EvaluationAutomated Software TestingSoftware Engineering BenchmarksMutation TestingSynthetic Benchmark GenerationAutomated Test Generation
AskPDF