Metamorphic Testing

Follow this topic Ask about this topic

Metamorphic testing studies whether a system behaves consistently when its inputs are changed in ways that should preserve or predictably alter the result. It is useful when expected outputs are hard to specify, and is applied to software, scientific and clinical models, and language or embodied AI systems. Approaches define relations from domain laws or specifications, or generate them from code and use transformed inputs to test APIs, programs, and models. A central challenge is validating each relation’s preconditions and limits: invalid relations can create false alarms, while even sound tests may miss errors that the chosen transformations cannot expose.

Latest papers

21 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses
Oct 21, 2026Vaidehi HegdeLanguage Model Reasoning Evaluation Language Model Error Detection
Indian Institute of Technology (Indian School of Mines) Dhanbad
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
Oct 20, 2026Ethan Ludwick, Yu-Chung HsiaoLLM-as-a-Judge Automated Test Generation
Cisco Systems · Anduril Industries
Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations
Sep 4, 2026FNU AditiSystemVerilog Assertion Generation Language Model Robustness
RotDroid: Cross-Orientation State Equivalence Testing for Detecting GUI Rotation Bugs in Android Apps
Aug 26, 2026Mengdi Qin, Bo JiangFault Detection Mobile GUI Automation
SKLCCSE Beihang University Beijing 100191, China
Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
Aug 6, 2026Rasul Khanbayov, Hasan KurbanVLM Evaluation Vision-Language Models
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents
Jul 26, 2026Gengyang Xu, Dongwei Xiao, Yiteng Peng +1Metamorphic Testing Embodied AI
Hong Kong University of Science and Technology, China
Metamorphic Testing for Clinical ML Models: A Framework Proposal and Pilot Study
Jul 25, 2026Jie JW Wu, Feiyu E, Bo ChenMetamorphic Testing Clinical Outcome Prediction
Michigan Technological University Houghton, MI, USA
Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
Jul 18, 2026Haifeng Li, Mo HaiMetamorphic Testing Formal Verification
School of Information, Central University of Finance and Economics
Domain-Validity-Gated Metamorphic Testing of Scientific ML Surrogates
Jun 16, 2026Meng Li, Xiaohua Yang, Jie Liu +1Metamorphic Testing Scientific ML
School of Computing, University of South China, Hengyang, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, China
Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning
Jun 4, 2026Helge Spieker, Jørn Eirik Betten, Arnaud GotliebMetamorphic Testing Explainability Evaluation
Simula Research Laboratory Oslo, Norway
Multi-Agent LLM-based Metamorphic Testing for REST APIs
May 27, 2026Shehroz Khan, Abdullah Mughees, Gaadha Sudheerbabu +2Automated Software Testing Metamorphic Testing
Abo Akademi University Turku, Finland
Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations
May 24, 2026Ashir Kulshreshtha, Abdullah Mughees, Gaadha Sudheerbabu +4Automated Software Testing Metamorphic Testing
Abo Akademi University, Finland · University of Turku, Finland · Novia University of Applied Sciences, Finland
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
May 19, 2026Quanxing Xu, Yuhao Tian, Ling Zhou +4Visual Question Answering Automated Software Testing
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3
A semantic mutation metric for metamorphic relation adequacy in scientific computing programs
May 17, 2026Meng Li, Xiaohua Yang, Jie Liu +1Automated Software Testing Metamorphic Testing
School of Computing, University of South China, Hengyang, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, China
NOETHER: A Constructive Framework for Metamorphic Pattern Discovery from Operator Algebras
May 17, 2026Meng Li, Xiaohua Yang, Jie Liu +1Automated Software Testing Metamorphic Testing
School of Computing, University of South China, Hengyang, 421001, China · Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China · CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China +2
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
May 12, 2026Zenghui Zhou, Man Li, Xiaoke Fang +3LLM Evaluation Automated Software Testing
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair
Apr 23, 2026Milan De Koning, Ali Asgari, Pouria Derakhshanfar +1LLM Evaluation Metamorphic Testing
JetBrains Research, Amsterdam, The Netherlands · Delft University of Technology, Delft, The Netherlands
MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis
Apr 11, 2026Congying Xu, Hengcheng Zhu, Songqiang Chen +3Metamorphic Testing Automated Test Generation
The Hong Kong University of Science and Technology, Hong Kong, China and Guangzhou HKUST Fok Ying Tung Research Institute, Guangzhou, China · University of Auckland, Auckland, New Zealand
From Untestable to Testable: Metamorphic Testing in the Age of LLMs
Mar 25, 2026Valerio TerragniAutomated Software Testing Metamorphic Testing
University of Auckland
Semantic Invariance in Agentic AI
Mar 13, 2026I. de Zarzà, J. de Curtò, Jordi Cabot +2AI Agent Reliability Language Model Robustness
Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg · Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain +2
Metamorphic Testing of Vision-Language Action-Enabled Robots
Feb 26, 2026Pablo Valle, Sergio Segura, Shaukat Ali +1Metamorphic Testing Vision-Language-Action Models
Mondragon University, Mondragon, Spain · University of Seville, Seville, Spain · Simula Research Laboratory, Oslo, Norway

Oct 21, 2026J/K move · Enter open · S save

Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses

Vaidehi Hegde

Indian Institute of Technology (Indian School of Mines) Dhanbad

Language Model Reasoning Evaluation Language Model Error Detection Metamorphic Testing

Open paper AskPDF arXiv

When a language model is deployed, no reference solution exists, so the reasoning trace it writes is the only evidence for judging its answer. The reference-free substitutes each fail differently: self-assessment degrades the answer it was meant to protect, sampling agreement measures stability rather than correctness, and perturbation studies read the expected answer off a human-written template, grading a model rather than a trace. We ask how far a single trace can be verified with no reference solution, no judge model and no weight update, and find, for a frozen LLaMA-2-7B-chat on GSM8K, that detection is not what limits such a pipeline. Checking a trace against itself reaches an AUROC of 0.741 and stops there for a structural reason: two thirds of wrong traces state no false equation, having executed a mistaken plan correctly, and on wrong-plan failures the detector flags a wrong trace less often than a right one. A candidate-derived metamorphic probe exposes many of these failures: we abstract the trace into a program over the question's quantities, perturb one quantity, execute the program, and ask the model the rewritten question once, so every trace supplies its own test. Valid perturbations exist for 937 of 1,319 test questions (71.0%); among the 847 that also have a parseable first trace, supported traces are correct 60.6% of the time against a 31.6% base rate, although the probe also refutes 45.5% of correct traces. No tested policy yields a statistically significant improvement in accuracy. A flag earns its retry only when detector precision exceeds D/(R+D), where R and D are the rates at which a retry repairs a wrong trace and damages a correct one; our repair rate is so low that, at the same retry budget, even a perfect detector would recover an estimated 66 of 1,319 questions. Repair, not detection, is the open problem.