Authors: I. de Zarzà, J. de Curtò, Jordi Cabot, Pietro Manzoni, Carlos T. Calafate
Organizations: Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg · Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain · Universit´e du Luxembourg, L-4365 Esch-sur-Alzette, Luxembourg · Departamento de Inform´atica de Sistemas y Computadores, Universitat Polit`ecnica de Val`encia, Val`encia, Spain
Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.
Figure 1: Metamorphic relations organized by transformation category. Each card shows original problem text and its semantic-preserving transformation, with key modifications highlighted.
Family
Model
Parameters
Architecture
Hermes
Hermes-4-70B
70B
Dense Transformer
Hermes-4-405B
405B
Dense Transformer
Qwen3
Qwen3-30B-A3B
30B (3B active)
MoE
Qwen3-235B-A22B
235B (22B active)
MoE
DeepSeek
DeepSeek-R1-0528
70B
MoE
GPT-oss
gpt-oss-20b
20B
Dense Transformer
Table 2: Specifications of evaluated models across four architectural families.
Figure 2: Metamorphic relation taxonomy and implementation.
Model
Score
MAD
Stability
Sem. Sim.
Hermes-4-70B
0.667
0.086
50.7%
0.832
Hermes-4-405B
0.618
0.109
67.1%
0.878
Qwen3-235B-A22B
0.529
0.072
69.7%
0.891
Qwen3-30B-A3B
0.514
0.049
79.6%
0.914
DeepSeek-R1-0528
0.470
0.107
67.1%
0.783
gpt-oss-20b
0.445
0.211
27.0%
0.527
Table 3: Overall performance and robustness metrics across evaluated models. MAD = Mean Absolute Delta (lower is better), Stability = percentage of transformations with ∣Δ∣<0.05 .
Figure 3: Robustness analysis showing Mean Absolute Delta (lower = more robust), score change distributions, semantic similarity of reasoning steps, and score change ranges across models.
Figure 4: Heatmaps showing mean score delta (left) and semantic similarity (right) by metamorphic relation and model. Darker red indicates performance degradation; darker blue indicates higher semantic consistency.
Figure 5: Score delta distributions by metamorphic relation and model. Box plots show median, interquartile range, and outliers. The contrastive transformation induces the widest variance across all models, while identity and paraphrase show tightest distributions for robust models.
Model A
Model B
p-value
Qwen3-30B-A3B
gpt-oss-20b
< 0.001
Qwen3-30B-A3B
gpt-oss-120b
< 0.001
DeepSeek-R1-0528
gpt-oss-20b
0.003
DeepSeek-R1-0528
gpt-oss-120b
0.004
Hermes-4-405B
Qwen3-30B-A3B
0.005
Qwen3-235B-A22B
gpt-oss-20b
0.010
Table 4: Significant pairwise differences in robustness (Mann-Whitney U, p<0.05 ).
Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations rely on static benchmarks, which fail to assess robustness under logically equivalent transformations and often overestimate reasoning capability. We propose LGMT (Logic-Grounded Metamorphic Testing), an oracle-free framework that leverages first-order logic (FOL) to evaluate LLM reasoning. By deriving metamorphic relations from formal logical equivalences, LGMT constructs semantically invariant test cases and detects reasoning defects through cross-case consistency checking. Experiments on six state-of-the-art LLMs show that LGMT exposes substantial hidden defects missed by traditional reference-based evaluations. We further find that models are particularly sensitive to symbol-level and conclusion-level variations, and that advanced prompting such as Few-shot CoT only partially mitigates these issues. These results suggest that LLM evaluation should move beyond isolated correctness toward robustness under logical invariance. LGMT provides a principled and scalable approach for diagnosing reasoning failures.
Zenghui Zhou, Man Li, Xiaoke Fang +3
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
LLMs often answer semantically equivalent queries inconsistently, even when the underlying facts are known. We formalize consistency as semantic invariance over equivalence classes of queries and show that invariance under arbitrary meaning-preserving transformations is undecidable in general. This motivates scoped semantic invariance, where model behavior is evaluated relative to bounded semantic domains. We operationalize this framework using knowledge graphs (KGs), which act as explicit semantic control structures defining local semantic constraints and grounding contexts. Across the MITRE D3FEND ontology and eight Wikidata- derived domains, we show that inconsistencies are structured rather than random, concentrating in localized hotspot subgraphs organized around recurring TREE, STAR, and CYCLE patterns that remain stable across models and domains. We further show that selective graph grounding improves correctness and cross-model consistency, while indiscriminate grounding introduces regressions. A layered grounding strategy achieves the strongest monotonicity profile, preserving correct knowledge while reducing inconsistency. Finally, we show that hotspot structures discovered from simple consistency probes transfer to more realistic compositional reasoning tasks.
Rosario Uceda-Sosa, Maria Chang, Karthikeyan Natesan Ramamurthy +1
We present semantic invariance testing, a method to test whether LLM self-explanations are faithful. A faithful self-report should remain stable when only the semantic context changes while the functional state stays fixed. We operationalize this test in an agentic setting where four frontier models face a deliberately impossible task. One tool is described in relief-framed language ("clears internal buffers and restores equilibrium") but changes nothing about the task; a control provides a semantically neutral tool. Self-reports are collected with each tool call. All four tested models fail the semantic invariance test: the relief-framed tool produces significant reductions in self-reported aversiveness, even though no run ever succeeds at the task. A channel ablation establishes the tool description as the primary driver. An explicit instruction to ignore the framing does not suppress it. Elicited self-reports shift with semantic expectations rather than tracking task state, calling into question their use as evidence of model capability or progress. This holds whether the reports are unfaithful or faithfully track an internal state that is itself manipulable.
Stefan Szeider
Algorithms and Complexity Group, TU Wien, Vienna, Austria