Automated Test Generation

Follow this topic Ask about this topic

Automated test generation studies how to create inputs and checks that reveal defects in software, APIs, hardware, and simulated systems. Methods include search-based testing and fuzzing, symbolic and constraint-based generation, property-based and metamorphic testing, and increasingly LLM agents guided by execution, coverage, or mutation feedback. Evaluation commonly measures coverage, mutation scores, bug detection, and whether tests distinguish correct behavior from faulty implementations. A central challenge is that executable tests and high coverage do not guarantee semantic correctness: weak oracles, incomplete specifications, and prompts based on buggy code can produce tests that miss faults or reinforce them.

Latest papers

133 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Oct 27, 2026Binhang Qi, Yiming Liu, Yun Lin +3Automated Test Generation LLM Agent Evaluation
National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1
From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science
Oct 21, 2026Yifeng He, Jiachen Liu, Hao ChenProperty-Based Testing Scientific Claim Verification
University of California, Davis · ARA Lab · The University of Hong Kong
FaultGene: LLM-Assisted Fault-Augmented Test Generation for Hardware-in-the-Loop Validation of Autonomous Driving Systems
Oct 21, 2026Chaoran Yuan, Fadi KurdahiAutomated Test Generation Autonomous Driving Safety Evaluation
Center for Embedded and Cyber-Physical Systems, University of California, Irvine, Irvine, USA
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
Oct 20, 2026Ethan Ludwick, Yu-Chung HsiaoLLM-as-a-Judge Automated Test Generation
Cisco Systems · Anduril Industries
TEX: Test-Time Scaling Testing Agents via Execution-based Cross-Validation
Oct 19, 2026Akash Gokul, Srijan Bansal, Yingbo Zhou +1Test-Time Scaling Software Engineering Agents
NVIDIA · Salesforce Research · Recursive
Certifying Repeatable Entry-Replay Witnesses for COBOL-to-Java Migration
Oct 19, 2026Andras L. Ferenczi, Mariya Bessonov, Allwyn Ponnudurai +3Software Engineering Automated Test Generation
American Express
FuzzInsight: Navigating the Efficiency-Effectiveness Trade-off in Coverage-Guided DNN Fuzzing
Oct 16, 2026Vivek Vekariya, Fathiyeh Faghih, Alexander PretschnerAutomated Test Generation Neural Network Robustness
Technical University of Munich, Germany · Birmingham City University, United Kingdom · Technical University of Munich & fortiss GmbH, Germany
SWEPatchTest: Evaluating Software Tests Beyond Reproduction
Oct 13, 2026Hua Lu, Pengcheng Wang, Han Li +4Software Engineering Benchmarks Automated Test Generation
Baidu Inc. · Nanjing University · The Chinese University of Hong Kong
SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
Small but Mighty: Test Suites as Improvement Signals for Self-Evolving Agents
Oct 11, 2026Yutong Cheng, Wei Cheng, Wenchao Yu +2Automated Test Generation LLM Agent Self-Improvement
Virginia Tech · NEC Laboratories America
Grounding Deep Learning Framework Testing in Data Flow
Oct 11, 2026Jiatao Ge, Yanzhou Mu, Yinglong Zou +3Automated Software Testing DNN Testing
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · UNIST, Ulsan, Republic of Korea
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Oct 11, 2026Jingjing Liu, Dishu Yang, Hongbo Qin +1Code Generation Code Generation Evaluation
University of California, Berkeley, Berkeley, CA, USA · Northeastern University, San Jose, CA, USA · Northeastern University, Boston, MA, USA
Targeting the Right Situations: Mutation Testing of Fuzzy-Powered Situation Hyperspace
Oct 10, 2026Nawshin Mannan PromaAutomated Software Testing Robot Safety
University of York, Heslington, United Kingdom
QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
Oct 9, 2026Meriem Terki, Nouhaila Innan, Minghao Shao +3Automated Test Generation LLM Evaluation
Ecole Supérieure en Informatique de Sidi Bel Abbès, Algeria · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE +1
QuSema: Detecting Silent Bugs in Quantum Libraries via Quantum-knowledge-enhanced Agents
Oct 7, 2026Yujin Song, Kaining Zhang, Qixin Zhang +3Fault Detection Automated Software Testing
Nanyang Technological University, Singapore · Hong Kong University of Science and Technology, Hong Kong, China · Zhejiang University of Technology, Hangzhou, China
TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble
Oct 7, 2026Pengfei He, Jiayuan Zhou, Shaowei Wang +1Automated Test Generation Automated Software Testing
University of Manitoba, Winnipeg, Canada · Huawei Canada, Markham, Canada · Huawei Technologies, Hangzhou, China
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Oct 7, 2026Shuangjie Yao, Hao Wang, Koushik Sen +3LLM Agent Evaluation AI Agent Benchmarks
Columbia University · University of California, Berkeley · George Mason University
Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
Oct 4, 2026Yifan Xiong, Jingyi Ge, Zhenpeng Chen +1LLM Agent Harnesses Automated Test Generation
University of Illinois Urbana-Champaign, USA · University of California, Berkeley, USA · Tsinghua University, China
GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing
Oct 2, 2026Masahiro KatoAutomated Software Testing AI Coding Agents
Mizuho-DL Financial Technology Co., Ltd.
AuraForge: Scaling Security Supervision for Training Coding Agents
Oct 1, 2026Danqing Wang, Songwen Zhao, Harsh Sharma +4AI Coding Agents Software Security
Carnegie Mellon University · University of California, Los Angeles · ScOp Venture Capital.
Improving Large Language Models for Code through Runtime Program-State Reasoning
Sep 28, 2026Hongwei Li, Spandan Garg, Yufan HuangAutomated Program Repair Automated Test Generation
University of California, Santa Barbara · Microsoft
Scenario-Driven Neuroevolution: Using Models to Guide Test Generation for Games
Sep 23, 2026Gijs van Cuyck, Patric Feldmeier, Jan Tretmans +1Automated Software Testing Automated Test Generation
Radboud University, Institute iCIS, Nijmegen, The Netherlands · University of Passau, Germany · TNO-ESI, Eindhoven, The Netherlands
Probabilistic Modelling of Operational Design Domains, A New Approach for Testing AI Systems
Sep 21, 2026Hans-Werner WiesbrockProbabilistic Graphical Models Knowledge Representation
ITPower Solutions GmbH Kolonnenstrasse 26 10829 Berlin, Germany
SyzHarness: Patch-Based Kernel Bug Reproduction with LLM-Synthesized Fuzzing Harnesses
Sep 20, 2026Xingyu Li, Juefei Pu, Haonan Li +4Software Vulnerability Detection LLM-Based Program Synthesis
University of California, Riverside
Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation
Sep 18, 2026Ana Nunez, Peyman NajafiradRL for Code Generation Code Generation
Secure AI and Autonomy Lab University of Texas at San Antonio
CovR: Coverage-Aware Hardware Verification via Reasoning-Guided Reinforcement Learning
Sep 15, 2026Manar Abdelatty, Maryam Nouh, Sherief RedaAutomated Test Generation Reinforcement Learning with Verifiable Rewards
School of Engineering Brown University Providence, RI, USA
Confidence-Gated Transductive Test Generation for Code Reranking
Sep 11, 2026Sungjae Lee, Youngsik Yoon, Seockbean Song +3LLM Reranking Automated Test Generation
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China
ExecCritic: Learn to Test, Test to Improve for Coding Agents
Sep 8, 2026Leitian Tao, Baolin Peng, Haorui Wang +7RL for Code Generation Software Engineering Agents
University of Wisconsin–Madison · Microsoft Research · Georgia Tech
Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs
Sep 3, 2026Jiacheng Xu, Wentao Zhang, Zhiyi Lyu +4Automated Test Generation LLM Code Generation
Nanyang Technological University, Singapore · Skywork AI
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Aug 27, 2026Zeming Liu, Hang Lyu, Jingtao ZhangAutomated Software Testing Mutation Testing
Independent Researchers

Oct 27, 2026J/K move · Enter open · S save

Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements

Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou+2

National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China+1

Automated Test Generation LLM Agent Evaluation Specification-Based Testing AI Agent Benchmarks

Open paper AskPDF arXiv

As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.