Automated Test Generation

Follow this topic Ask about this topic

Automated test generation studies how to create inputs and checks that reveal defects in software, APIs, hardware, and simulated systems. Methods include search-based testing and fuzzing, symbolic and constraint-based generation, property-based and metamorphic testing, and increasingly LLM agents guided by execution, coverage, or mutation feedback. Evaluation commonly measures coverage, mutation scores, bug detection, and whether tests distinguish correct behavior from faulty implementations. A central challenge is that executable tests and high coverage do not guarantee semantic correctness: weak oracles, incomplete specifications, and prompts based on buggy code can produce tests that miss faults or reinforce them.

Latest papers

133 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

HAVEN: Hybrid Automated Verification ENgine for UVM Testbench Synthesis with LLMs
Apr 30, 2026Chang-Chih Meng, Yu-Ren Lu, Guan-Yu Lin +3Electronic Design Automation LLM-Based Program Synthesis
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
Apr 28, 2026Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3Automated Software Testing Benchmark Design
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1
Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents
Apr 28, 2026Jianming Chen, Yawen Wang, Junjie Wang +4Agent Failure Analysis AI Agent Evaluation
Institute of Software, Chinese Academy of Sciences, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China +2
Ethics Testing: Proactive Identification of Generative AI System Harms
Apr 23, 2026Shin Hwei Tan, Haibo Wang, Heng LiAI Safety Evaluation Automated Test Generation
Concordia University, Montreal, Canada · Polytechnique Montreal, Montreal, Canada
Call-Chain-Aware LLM-Based Test Generation for Java Projects
Apr 23, 2026Guancheng Wang, Qinghua Xu, Lionel C. Briand +2Automated Test Generation
Research Ireland Lero Centre for Software and University of Limerick, Ireland · University of Ottawa, Canada · The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China +1
PrismaDV: Automated Task-Aware Data Unit Test Generation
Apr 23, 2026Hao Chen, Arnab Phani, Sebastian SchelterAutomatic Prompt Optimization Automated Test Generation
BIFOLD & TU Berlin
You Don't Need Public Tests to Generate Correct Code
Apr 23, 2026Kaushitha Silva, Srinath PereraExecution-Guided Code Generation Code Generation
WSO2 Santa Clara, CA, USA
From Particles to Perils: SVGD-Based Hazardous Scenario Generation for Autonomous Driving Systems Testing
Apr 20, 2026Linfeng Liang, Xiao Cheng, Tsong Yueh Chen +1Autonomous Driving Safety Evaluation Autonomous Driving Simulation
Macquarie University, Australia · Swinburne University of Technology, Australia
Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics
Apr 20, 2026Khang Tran, Khoa Nguyen, Cristian Borcea +1LLM-Based Program Synthesis Automated Test Generation
New Jersey Institute of Technology, Newark, NJ, USA
ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimization
Apr 17, 2026Yixu Huang, Xinglei Yu, Zhongyu WeiExecution-Guided Code Generation Language Model Self-Improvement
School of Data Science Fudan University Shanghai, China
LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB
Apr 15, 2026Vekil Bekmyradov, Noah C. Pütz, Thomas Bartz-BeielsteinShortcut Learning in Language Models LLM Evaluation
THK-AI Research Cluster, TH Köln
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
Apr 13, 2026Zijie Zhao, Chenyuan Yang, Weidong Wang +3Automated Software Testing Software Engineering Agents
University of Illinois Urbana-Champaign, Champaign, USA
Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models
Apr 12, 2026Mehmet Can Şakiroğlu, H. Altay Güvenir, Kamer KayaMultiple-Choice Question Generation Automated Test Generation
Computer Engineering Department, Bilkent University, Ankara, Turkey · Faculty of Engineering and Natural Sciences, Sabanci University · VERIM, Center of Excellence in Data Analytics, Sabanci University
MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis
Apr 11, 2026Congying Xu, Hengcheng Zhu, Songqiang Chen +3Metamorphic Testing Automated Test Generation
The Hong Kong University of Science and Technology, Hong Kong, China and Guangzhou HKUST Fok Ying Tung Research Institute, Guangzhou, China · University of Auckland, Auckland, New Zealand
MIMIC-Py: An Extensible Tool for Personality-Driven Automated Game Testing with Large Language Models
Apr 9, 2026Yifei Chen, Sarra Habchi, Lili WeiAutomated Game Testing Automated Software Testing
Electrical and Computer Engineering, McGill University, Montréal, Québec, Canada · Cohere, Montréal, Québec, Canada
An empirical study of LoRA-based fine-tuning of large language models for automated test case generation
Apr 8, 2026Milad Moradi, Ke Yan, David Colwell +1Automated Test Generation LLM Fine-Tuning
AI Research Lab, Tricentis, Vienna, Austria · AI Research Lab, Tricentis, Sydney, Australia
ExplainFuzz: Explainable and Constraint-Conditioned Test Generation with Probabilistic Circuits
Apr 8, 2026Annaëlle Baiget, Jaron Maene, Seongmin Lee +3Automated Test Generation Constrained Generative Modeling
UCLA, USA · KU Leuven, Belgium
Planning to Explore: Curiosity-Driven Planning for LLM Test Generation
Apr 6, 2026Alfonso Amayuelas, Firas Laakom, Piotr Piękos +5Automated Test Generation LLM Planning
University of California, Santa Barbara · King Abdullah University of Science and Technology
AI-Assisted Unit Test Writing and Test-Driven Code Refactoring: A Case Study
Apr 3, 2026Ema Smolic, Mario Brcic, Luka Hobor +1Automated Code Refactoring AI-Assisted Software Development
It from Bit d.o.o., Zagreb, Croatia · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia
BACE: LLM-based Code Generation through Bayesian Anchored Co-Evolution of Code and Test Populations
Mar 30, 2026Kaushitha Silva, Srinath PereraLLM Code Generation LLM-Based Program Synthesis
WSO2 Santa Clara, CA, USA
WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
Mar 26, 2026Fanheng Kong, Jingyuan Zhang, Yang Yue +10Automated Software Testing Computer-Use Agent Benchmarks
Northeastern University · Kuaishou Technology
UCAgent: An End-to-End Agent for Block-Level Functional Verification
Mar 26, 2026Junyue Wang, Zhicheng Yao, Yan Pi +6LLM Agent Verification Automated Test Generation
State Key Lab of Processors, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Beijing Institute of Open Source Chip
LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops
Mar 24, 2026Ravin Ravi, Dylan Bradshaw, Stefano Ruberto +2LLM Code Generation LLM Self-Refinement
University of Auckland, Auckland, New Zealand · JRC European Commission, Ispra, Italy · King’s College London, London, United Kingdom
Evaluating LLM-Based Test Generation Under Software Evolution
Mar 24, 2026Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali GulzarAutomated Test Generation Automated Software Testing
Virginia Tech, USA · Carnegie Mellon University, USA
LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search
Mar 23, 2026Yujia Chen, Yingli Zhou, Fangyuan Zhang +1Automated Test Generation Relational Databases
Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, Shenzhen, China · Huawei Hong Kong Research Center, Hongkong, China
VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?
Mar 16, 2026Srijan Bansal, Jiao Fangkai, Yilun Zhou +3LLM Self-Correction Language Model Error Detection
Salesforce AI Research · Nanyang Technological University · A*STAR
Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning
Mar 16, 2026Aozhe Wang, Yuchen Yan, Nan Zhou +5LLM Code Generation RL for Code Generation
Zhejiang University
STADA: Specification-based Testing for Autonomous Driving Agents
Mar 11, 2026Joy Saha, Trey Woodlief, Sebastian Elbaum +1Specification-Based Testing Automated Test Generation
University of Virginia, Virginia, USA · William & Mary, Virginia, USA
Declarative Scenario-based Testing with RoadLogic
Mar 10, 2026Ezio Bartocci, Alessio Gambi, Felix Gigler +2Autonomous Driving Scenario Generation Automated Test Generation
TU Wien Vienna, Austria · AIT Austrian Institute of Technology Vienna, Austria
MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning
Mar 2, 2026Sicheng Zhu, Jiajun Wang, Jiawei Ai +1Automated Test Generation Mutation Testing
Fudan University, Shanghai, China · University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China +1

Apr 30, 2026J/K move · Enter open · S save

HAVEN: Hybrid Automated Verification ENgine for UVM Testbench Synthesis with LLMs

Chang-Chih Meng, Yu-Ren Lu, Guan-Yu Lin, Tsung Tai Yeh+2

National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Electronic Design Automation LLM-Based Program Synthesis Agentic Code Generation Automated Test Generation

Open paper AskPDF arXiv

Integrated Circuit (IC) verification consumes nearly 70% of the IC development cycle, and recent research leverages Large Language Models (LLMs) to automatically generate testbenches and reduce verification overhead. However, LLMs have difficulty generating testbenches correctly. Unlike high-level programming languages, Hardware Description Languages (HDLs) are extremely rare in LLMs training data, leading LLMs to produce incorrect code. To overcome challenges when using LLMs to generate Universal Verification Methodology (UVM) testbenches and sequences, wepropose HAVEN (Hybrid Automated Verification ENgine) to prevent LLMs from writing HDL directly. For UVM testbench generation, HAVEN utilizes LLM agents to analyze design specifications to produce a structured architectural plan. The HAVEN Template Engine then combines with predefined and protocol-specific templates to generate all UVM components with correct bus-handshake timing. For UVM sequence generation, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types. A set of predefined DSL patterns first establishes sequences that achieve a high coverage rate without LLM involvement. HAVEN continues to improve the coverage rate by iteratively leveraging LLM agents to analyze coverage gap reports and compose additional targeted DSL sequences. Unlike previous works, HAVEN is the first system that utilizes pre-defined, protocol-specific Jinja2 templates to generate all UVM components and UVM sequences using our proposed Protocol-Aware DSL and rule-based code generator. Our experimental results on 19 open-source IP designs spanning three interface protocols (Direct, Wishbone, AXI4-Lite) show that HAVEN achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage on average, and is SOTA among LLM-assisted testbench generation systems.