Mutation Testing

Follow this topic Ask about this topic

Mutation testing deliberately changes programs, models, or system behavior to check whether a test suite detects the resulting faults. Researchers use rule-based, data-flow, semantic, and LLM-generated mutations, alongside targeted test generation, to assess software, APIs, machine-learning systems, and other complex tools. Benchmarks such as HumanEval and MBPP help compare test suites, while statistical methods address nondeterministic settings such as AI agents and quantum models. Mutation scores can guide testing, but their ability to predict detection of real bugs depends on how realistic and representative the mutations are.

Latest papers

18 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

VLTest: Finding Joint Image-Text Failures in Vision-Language Models
Oct 22, 2026Bin Duan, Jintao Lin, Guowei YangVLM Evaluation Mutation Testing
The University of Queensland, Brisbane, Australia
Grounding Deep Learning Framework Testing in Data Flow
Oct 11, 2026Jiatao Ge, Yanzhou Mu, Yinglong Zou +3Automated Software Testing DNN Testing
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · UNIST, Ulsan, Republic of Korea
Targeting the Right Situations: Mutation Testing of Fuzzy-Powered Situation Hyperspace
Oct 10, 2026Nawshin Mannan PromaAutomated Software Testing Robot Safety
University of York, Heslington, United Kingdom
Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites
Oct 2, 2026Xin Xu, Siru TaoAutomated Software Testing AI Agent Monitoring
Carnegie Mellon University Eragon
Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models
Sep 24, 2026Nils Kiele, Zainab Saad, Zirui Wang +2Automated Software Testing Mutation Testing
Department of Electrical and Software Engineering, University of Calgary, AB, Canada
SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
Sep 8, 2026Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1Mutation Testing Text-to-SQL
Polytechnique Montréal
Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
Sep 2, 2026Mingzhe Du, Anh Tuan Luu, Dong Huang +1GPU Kernel Correctness Verification Benchmark Auditing
National University of Singapore · Nanyang Technological University · CAIR, VinUniversity
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Aug 27, 2026Zeming Liu, Hang Lyu, Jingtao ZhangAutomated Software Testing Mutation Testing
Independent Researchers
Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
Jul 25, 2026Jeff OttersonAutomated Software Testing LLM Auditing
W. P. Carey School of Business, Arizona State University
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
Jul 24, 2026Junda Zhao, Shurui Zhou, Eldan CohenLLM Evaluation Automated Software Testing
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
May 21, 2026Yuxuan Sun, Yuze Zhao, Yufeng Wang +6LLM Evaluation Automated Software Testing
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3
A semantic mutation metric for metamorphic relation adequacy in scientific computing programs
May 17, 2026Meng Li, Xiaohua Yang, Jie Liu +1Automated Software Testing Metamorphic Testing
School of Computing, University of South China, Hengyang, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, China
Efficient Mutation Testing of Quantum Machine Learning Models
Apr 30, 2026Emma Andrews, Prabhat MishraAutomated Software Testing Mutation Testing
University of Florida Gainesville, Florida, USA
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
Apr 28, 2026Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3Automated Software Testing Benchmark Design
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1
QuanForge: A Mutation Testing Framework for Quantum Neural Networks
Apr 22, 2026Minqi Shao, Shangzhou Xia, Jianjun ZhaoAutomated Software Testing Mutation Testing
Kyushu University, Fukuoka, Japan
AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
Mar 3, 2026Varun Pratap BhardwajAI Agent Evaluation Mutation Testing
Independent Researcher
MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning
Mar 2, 2026Sicheng Zhu, Jiajun Wang, Jiawei Ai +1Automated Test Generation Mutation Testing
Fudan University, Shanghai, China · University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China +1
UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software
Feb 23, 2026Chengjie Lu, Jiahui Wu, Shaukat Ali +5Mutation Testing Automated Software Testing
Simula Research Laboratory and University of Oslo · Simula Research Laboratory · Danish Technological Institute +1

Oct 22, 2026J/K move · Enter open · S save

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan, Jintao Lin, Guowei Yang

The University of Queensland, Brisbane, Australia

VLM Evaluation Mutation Testing Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.