LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

Zenghui Zhou; Xiaoke Fang; Zheng Zheng

cs.AIOct 23, 2026

LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

Authors: Zenghui Zhou, Xiaoke Fang, Zheng Zheng

Organizations: School of Automation Science and Electrical Engineering, Beihang University Xueyuan Road, Haidian District, Beijing, 100191, China

Abstract

Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsistencies. To address this limitation, we present LGMT, an open-source tool that detects reasoning inconsistencies through metamorphic relations grounded in first-order logic. Given a source test case in premise-conclusion form, LGMT generates logically equivalent follow-up cases and checks whether the model's entailment predictions agree across the pair, without ground-truth labels. The tool provides 20 built-in metamorphic relations, supports custom ones verified by the Z3 theorem prover, and runs from a CLI or a web studio, including an offline mock mode requiring no API key. Across six widely used LLMs on a bundled benchmark of 3,091 metamorphic groups, LGMT reveals inconsistency rates ranging from 20.84% to 29.44%. LGMT is publicly available on GitHub at https://github.com/MaxwelsDonc/lgmt-tool, with a demonstration video at https://youtu.be/gou4opptu1U.

Figures & tables

Fig. 1: The LGMT workflow. Five stages turn source test cases into paired queries: one of 20 MRs rewrites the logic, only the changed statements are realized in natural language, and both cases are queried and compared. The tool ships a 3,091-case benchmark (plus a 40-case demo subset) and targets any OpenAI-compatible model.

Fig. 2: A violation card for MR-C3 in LGMT’s HTML report. The card shows the four-column FOL/NL diff and the side-by-side reasoning traces that localize where and why the model’s prediction diverges.

Model	Acc static	Acc cons	MVR	HDR
GPT-5.2	73.34%	56.84%	26.76%	16.50%
Claude 4.5 Sonnet	73.53%	60.97%	20.84%	12.56%
Llama 3.3 70B	65.38%	52.38%	25.43%	13.01%
DeepSeek Chat	69.91%	55.65%	24.72%	14.27%
o4-mini	69.36%	56.20%	22.06%	13.17%
DeepSeek Reasoner	72.44%	56.71%	29.44%	15.72%

TABLE I: Validation on the 3,091-Group Benchmark (Zero-Shot CoT). All six models exceed a 20% MVR, indicating substantial reasoning inconsistency that static accuracy misses.

Reasoning Consistency in Language Models

Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.

See all 122 papers

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
May 12, 2026Zenghui Zhou, Man Li, Xiaoke Fang +3LLM Evaluation Automated Software Testing
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
Controlled Reformulation Testing for Logical Consistency in Large Language Models
Jul 16, 2026Alexander Gu, Alan ChenLLM Evaluation Reasoning Consistency in Language Models
University of Texas at Austin
ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems
Jan 5, 2026Noel ThomasSymbolic Reasoning in Language Models LLM Reasoning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

May 12, 2026J/K move · Enter open · S save

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou+2

School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China

LLM Evaluation Automated Software Testing Metamorphic Testing LLM Reliability Language Model Robustness LLM Reasoning

Open paper AskPDF arXiv

Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations rely on static benchmarks, which fail to assess robustness under logically equivalent transformations and often overestimate reasoning capability. We propose LGMT (Logic-Grounded Metamorphic Testing), an oracle-free framework that leverages first-order logic (FOL) to evaluate LLM reasoning. By deriving metamorphic relations from formal logical equivalences, LGMT constructs semantically invariant test cases and detects reasoning defects through cross-case consistency checking. Experiments on six state-of-the-art LLMs show that LGMT exposes substantial hidden defects missed by traditional reference-based evaluations. We further find that models are particularly sensitive to symbol-level and conclusion-level variations, and that advanced prompting such as Few-shot CoT only partially mitigates these issues. These results suggest that LLM evaluation should move beyond isolated correctness toward robustness under logical invariance. LGMT provides a principled and scalable approach for diagnosing reasoning failures.

Abstract

Reasoning Consistency in Language Models

Explore similar work

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

Controlled Reformulation Testing for Logical Consistency in Large Language Models

ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems