From Untestable to Testable: Metamorphic Testing in the Age of LLMs

Valerio Terragni

cs.SEMar 25, 2026

From Untestable to Testable: Metamorphic Testing in the Age of LLMs

Authors: Valerio Terragni

Organizations: University of Auckland

Abstract

This article discusses the challenges of testing software systems with increasingly integrated AI and LLM functionalities. LLMs are powerful but unreliable, and labeled ground truth for testing rarely scales. Metamorphic Testing solves this by turning relations among multiple test executions into executable test oracles.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 135 papers

LLMORPH: Automated Metamorphic Testing of Large Language Models
Mar 24, 2026Steven Cho, Stefano Ruberto, Valerio TerragniLLM Evaluation Language Model Robustness
University of Auckland, Auckland, New Zealand · JRC European Commission, Ispra, Italy
Multi-Agent LLM-based Metamorphic Testing for REST APIs
May 27, 2026Shehroz Khan, Abdullah Mughees, Gaadha Sudheerbabu +2Automated Software Testing Metamorphic Testing
Abo Akademi University Turku, Finland
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
May 21, 2026Yuxuan Sun, Yuze Zhao, Yufeng Wang +6LLM Evaluation Automated Software Testing
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3

Mar 24, 2026J/K move · Enter open · S save

LLMORPH: Automated Metamorphic Testing of Large Language Models

Steven Cho, Stefano Ruberto, Valerio Terragni

University of Auckland, Auckland, New Zealand · JRC European Commission, Ispra, Italy

LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

Automated testing is essential for evaluating and improving the reliability of Large Language Models (LLMs), yet the lack of automated oracles for verifying output correctness remains a key challenge. We present LLMORPH, an automated testing tool specifically designed for LLMs performing NLP tasks, which leverages Metamorphic Testing (MT) to uncover faulty behaviors without relying on human-labeled data. MT uses Metamorphic Relations (MRs) to generate follow-up inputs from source test input, enabling detection of inconsistencies in model outputs without the need of expensive labelled data. LLMORPH is aimed at researchers and developers who want to evaluate the robustness of LLM-based NLP systems. In this paper, we detail the design, implementation, and practical usage of LLMORPH, demonstrating how it can be easily extended to any LLM, NLP task, and set of MRs. In our evaluation, we applied 36 MRs across four NLP benchmarks, testing three state-of-the-art LLMs: GPT-4, LLAMA3, and HERMES 2. This produced over 561,000 test executions. Results demonstrate LLMORPH's effectiveness in automatically exposing inconsistencies.

Abstract

Automated Software Testing

Explore similar work

LLMORPH: Automated Metamorphic Testing of Large Language Models

Multi-Agent LLM-based Metamorphic Testing for REST APIs

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?