Authors: I. de Zarzà, J. de Curtò, Jordi Cabot, Pietro Manzoni, Carlos T. Calafate
Organizations: Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg · Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain · Universit´e du Luxembourg, L-4365 Esch-sur-Alzette, Luxembourg · Departamento de Inform´atica de Sistemas y Computadores, Universitat Polit`ecnica de Val`encia, Val`encia, Spain
Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.
Figure 1: Metamorphic relations organized by transformation category. Each card shows original problem text and its semantic-preserving transformation, with key modifications highlighted.
Family
Model
Parameters
Architecture
Hermes
Hermes-4-70B
70B
Dense Transformer
Hermes-4-405B
405B
Dense Transformer
Qwen3
Qwen3-30B-A3B
30B (3B active)
MoE
Qwen3-235B-A22B
235B (22B active)
MoE
DeepSeek
DeepSeek-R1-0528
70B
MoE
GPT-oss
gpt-oss-20b
20B
Dense Transformer
Table 2: Specifications of evaluated models across four architectural families.
Figure 2: Metamorphic relation taxonomy and implementation.
Model
Score
MAD
Stability
Sem. Sim.
Hermes-4-70B
0.667
0.086
50.7%
0.832
Hermes-4-405B
0.618
0.109
67.1%
0.878
Qwen3-235B-A22B
0.529
0.072
69.7%
0.891
Qwen3-30B-A3B
0.514
0.049
79.6%
0.914
DeepSeek-R1-0528
0.470
0.107
67.1%
0.783
gpt-oss-20b
0.445
0.211
27.0%
0.527
Table 3: Overall performance and robustness metrics across evaluated models. MAD = Mean Absolute Delta (lower is better), Stability = percentage of transformations with ∣Δ∣<0.05 .
Figure 3: Robustness analysis showing Mean Absolute Delta (lower = more robust), score change distributions, semantic similarity of reasoning steps, and score change ranges across models.
Figure 4: Heatmaps showing mean score delta (left) and semantic similarity (right) by metamorphic relation and model. Darker red indicates performance degradation; darker blue indicates higher semantic consistency.
Figure 5: Score delta distributions by metamorphic relation and model. Box plots show median, interquartile range, and outliers. The contrastive transformation induces the widest variance across all models, while identity and paraphrase show tightest distributions for robust models.
Model A
Model B
p-value
Qwen3-30B-A3B
gpt-oss-20b
< 0.001
Qwen3-30B-A3B
gpt-oss-120b
< 0.001
DeepSeek-R1-0528
gpt-oss-20b
0.003
DeepSeek-R1-0528
gpt-oss-120b
0.004
Hermes-4-405B
Qwen3-30B-A3B
0.005
Qwen3-235B-A22B
gpt-oss-20b
0.010
Table 4: Significant pairwise differences in robustness (Mann-Whitney U, p<0.05 ).