Automated Software Testing

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 93

All topics

Multi-Agent LLM-based Metamorphic Testing for REST APIs
May 27, 2026Shehroz Khan, Abdullah Mughees, Gaadha Sudheerbabu +2Automated Software Testing Metamorphic Testing
Abo Akademi University Turku, Finland
HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML
May 26, 2026Jiajun Wu, Jian Yang, Tuney Zheng +4Automated Software Testing Automated Program Repair
1Beihang University · 2IQuest Research · 3Peking University
Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations
May 24, 2026Ashir Kulshreshtha, Abdullah Mughees, Gaadha Sudheerbabu +4Automated Software Testing Metamorphic Testing
Abo Akademi University, Finland · University of Turku, Finland · Novia University of Applied Sciences, Finland
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications
May 24, 2026Xiaoyue Lu, Xianglin Yang, Haijun Liu +4Automated Software Testing Language Model Safety Evaluation
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · National University of Singapore, Singapore · Independent Researcher
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
May 21, 2026Yuxuan Sun, Yuze Zhao, Yufeng Wang +6LLM Evaluation Automated Software Testing
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3
TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos
May 20, 2026Yakun Yu, Ashley Wiens, Adrián Barahona-Ríos +4VLM Evaluation Automated Software Testing
University of Alberta · Sony Interactive Entertainment
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
May 19, 2026Quanxing Xu, Yuhao Tian, Ling Zhou +4Visual Question Answering Automated Software Testing
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3
A semantic mutation metric for metamorphic relation adequacy in scientific computing programs
May 17, 2026Meng Li, Xiaohua Yang, Jie Liu +1Automated Software Testing Metamorphic Testing
School of Computing, University of South China, Hengyang, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, China
NOETHER: A Constructive Framework for Metamorphic Pattern Discovery from Operator Algebras
May 17, 2026Meng Li, Xiaohua Yang, Jie Liu +1Automated Software Testing Metamorphic Testing
School of Computing, University of South China, Hengyang, 421001, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China +2
Autonomous Intelligent Agents for Natural-Language-Driven Web Execution with Integrated Security Assurance
May 14, 2026Vinil Pasupuleti, Siva Rama Krishna Varma Bayyavarapu, Shrey TyagiAutomated Software Testing Software Security
International Business Machines (IBM) · South Carolina, United States · Docusign +4
Given, When, Then, Again: Mining Subscenario Refactoring Candidates in Behaviour-Driven Test Suites with ML Classifiers and LLM-Judge Baselines
May 14, 2026Ali Hassaan Mughal, Noor Fatima, Muhammad BilalAutomated Software Testing Automated Software Engineering
Independent Researcher; Applied MBA (Data Analytics), Texas Wesleyan University, Fort Worth, TX, USA. · Independent Researcher; B.E. Computer Engineering, National University of Sciences and Technology (NUST), Pakistan. · Independent Researcher; M.Sc. Management, Technical University of Munich, Munich, Germany.
LLM-Based Robustness Testing of Microservice Applications: An Empirical Study
May 13, 2026Hrushitha Goud Tigulla, Marco VieiraSoftware Engineering Automated Software Testing
College of Computing and Informatics University of North Carolina at Charlotte Charlotte, USA
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
May 13, 2026Lucas Jing, Xinqi Wang, Liao Zhang +1Automated Software Testing Software Engineering Agents
Tsinghua University · University of Washington · Beneficial AI Foundation +1
CA2: Code-Aware Agent for Automated Game Testing
May 13, 2026Valliappan Chidambaram Adaikkappan, Vincent Martineau, Joshua Romoff +1Automated Software Testing Reinforcement Learning
† McGill University · ⋄Mila Quebec AI Institute · ‡ Ubisoft
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
May 12, 2026Zenghui Zhou, Man Li, Xiaoke Fang +3LLM Evaluation Automated Software Testing
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
VISOR: A Vision-Language Model-based Test Oracle for Testing Robots
May 11, 2026Prasun Saurabh, Pablo Valle, Aitor Arrieta +2Uncertainty Estimation for VLA Models Automated Software Testing
Simula Research Laboratory and Oslo Metropolitan University · Mondragon University · National Institute of Informatics
Search-based Robustness Testing of Laptop Refurbishing Robotic Software
May 8, 2026Erblin Isaku, Hassan Sartaj, Shaukat Ali +2Automated Software Testing Adversarial Robustness
Simula Research Laboratory and University of Oslo · Simula Research Laboratory · Danish Technological Institute
GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection
May 8, 2026Chaobo Jia, Ruipeng Wan, Ting Sun +5Automated Software Testing Formal Verification
1CUHK · 2HUST · 3Lionrock AI Lab +3
Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents
May 4, 2026Reshabh K Sharma, Gaurav Mittal, Yu HuAutomated Software Testing Agent Evaluation
University of Washington, Seattle, WA, USA · Microsoft, Redmond, WA, USA
Practical Limits of Autonomous Test Repair: A Multi-Agent Case Study with LLM-Driven Discovery and Self-Correction
May 2, 2026Hyukjoo LeeAutomated Software Testing Software Engineering Agents
Independent Researcher Republic of Korea
FeedbackLLM: Metadata driven Multi-Agentic Language Agnostic Test Case Generator with Evolving prompt and Coverage Feedback
May 2, 2026Kushal Jasti, Tejamani Prashanth Sahu, Rishitha Pentyala +2Multi-Agent LLM Systems Automated Software Testing
Department of Computer Science and Engineering, SRM University AP, India
PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
May 1, 2026Gourisetty Venkata Sai Koushik, Dama Aditya, Mahankali Harish Sai +3Automated Software Testing LLM-Guided RL
Department of Computer Science and Engineering, SRM University AP
Efficient Mutation Testing of Quantum Machine Learning Models
Apr 30, 2026Emma Andrews, Prabhat MishraAutomated Software Testing Mutation Testing
University of Florida Gainesville, Florida, USA
Hierarchical Fault Detection and Diagnosis for Transformer Architectures
Apr 30, 2026Sigma Jahan, Saurabh Singh Rajput, Tushar Sharma +1Software Engineering Automated Software Testing
Dalhousie University, Halifax, Canada
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
Apr 28, 2026Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3Automated Software Testing Benchmark Design
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1
Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents
Apr 27, 2026M. MengAgent Failure Analysis Automated Software Testing
Shenzhen The Nine’s Light Technology Co., Ltd., Shenzhen, Guangdong, China
Efficient Agent Evaluation via Diversity-Guided User Simulation
Apr 23, 2026Itay Nakash, George Kour, Ateret Anaby-TavorAutomated Software Testing LLM Agent Evaluation
IBM Research
QuanForge: A Mutation Testing Framework for Quantum Neural Networks
Apr 22, 2026Minqi Shao, Shangzhou Xia, Jianjun ZhaoAutomated Software Testing Mutation Testing
Kyushu University, Fukuoka, Japan
Deja Vu at Scale: Paraphrase-Robust Detection of Duplicate Gherkin Steps in Behaviour-Driven Software Testing with Sentence-Transformer Embeddings and a 1.1M-Step Open Benchmark
Apr 22, 2026Ali Hassaan Mughal, Noor Fatima, Muhammad BilalAutomated Software Testing Benchmark Construction
Independent Researcher; Applied MBA (Data Analytics), Texas Wesleyan University, Fort Worth, TX, USA · Independent Researcher · Independent Researcher; B.E. Computer Engineering, National University of Sciences and Technology (NUST), Pakistan +1
Towards Reliable Testing of Machine Unlearning
Apr 16, 2026Anna Mazhar, Sainyam GalhotraData Leakage Automated Software Testing
Cornell University Ithaca, NY, USA
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
Apr 13, 2026Zijie Zhao, Chenyuan Yang, Weidong Wang +3Automated Software Testing Software Engineering Agents
University of Illinois Urbana-Champaign, Champaign, USA
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
Apr 1, 2026Alibek Kaliyev, Artem MaryanskyyAI Agent Reliability Tool-Use Evaluation
The University of Texas at Austin · Austin, Texas, USA · Uber +1
MASTEST: A LLM-Based Multi-Agent System For Testing RESTful APIs
Nov 22, 2025Xiaoke Han, Hong ZhuMulti-Agent LLM Systems Automated Software Testing
School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, UK
Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing
Mar 11, 2025Lev Sorokin, Matteo Biagiola, Andrea StoccoAutomated Software Testing Autonomous Driving Simulation
Technical University of Munich, Munich, Germany. · University of St. Gallen and Universit`a della Svizzera italiana, St. Gallen / Lugano, Switzerland. · fortiss GmbH, Munich, Germany.
Requirement-Based Testing: Enhancing Reinforcement Learning with Game Theory
Jul 26, 2024Ocan Sankur, Thierry Jéron, Nicolas Markey +2Automated Software Testing Automated Test Generation
DEVINE · MERCE-France · Mitsubishi Electric Corporation, Tokyo, Japan
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Date pendingZeming Liu, Hang Lyu, Jingtao ZhangAutomated Software Testing Mutation Testing
Independent Researchers

May 27, 2026, cs.SEJ/K move · Enter open · S save

Multi-Agent LLM-based Metamorphic Testing for REST APIs

Shehroz Khan, Abdullah Mughees, Gaadha Sudheerbabu, Tanwir Ahmad+1

Abo Akademi University Turku, Finland

Automated Software Testing Metamorphic Testing Automated Test Generation

Open paper AskPDF arXiv

As REST APIs become an increasingly significant part of software systems, their validation is becoming more critical. Hence, testing and uncovering underlying issues are of utmost importance for improving software quality. However, testing REST APIs is challenging mainly due to the difficulty of assessing whether the output of an API call is correct, i.e., the test oracle problem. Metamorphic testing is a specification-based testing approach for situations where correct outputs are unknown or not specified explicitly. To check the correctness of a system, relations between the different outputs are specified. We present ARMeta, a tool-supported approach that uses an LLM-based multi-agent workflow to support metamorphic testing of REST APIs documented with OpenAPI. The agentic workflow is used to identify metamorphic test scenarios and specify them in the Given-When-Then format. These scenarios are automatically implemented as executable tests and executed against the system under test. We evaluate ARMeta on two publicly available web applications that expose REST interfaces and compare its performance with a scenario-based testing baseline. The results show that ARMeta explores behaviors that serve as a complement to existing scenario-based testing approaches.

Momentum

Related topics

Latest papers 93