Competence-Based Evaluation of Logical Reasoning in Large Language Models

Jize Jiang; Chengxiang Zhai

cs.CLOct 16, 2026

Competence-Based Evaluation of Logical Reasoning in Large Language Models

Authors: Jize Jiang, Chengxiang Zhai

Organizations: University of Illinois Urbana-Champaign

Abstract

Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks. Such a performance-based evaluation methodology says little about a model's generalization capacity in reasoning. To address this limitation, we argue for a different evaluation paradigm, called competence-based evaluation (CBE), which tests whether models reliably apply the same competence across a family of problems. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer-preserving presentations test correctness beyond one formulation. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held-out transfer. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models.

Figures & tables

Panel of Figure 1 — Figure 1: Ordering generation and paired evaluation (illustrative instance). A latent order supplies adjacent premises and a gold endpoint answer. Equivalent prompts swap arguments with converse phrasing and reshuffle names and premises, preserving the context and query. Paired answers determine both-correct ( $R$ ), both-wrong ( $D$ ), or exactly-one-correct ( $U$ ) outcomes; repetition across instances and chain lengths yields the competence profile.

	front/behind					left/right
Model	Acc.Org	Acc.Eqv	Inv.	$R$	$\bar{\kappa}$	Acc.Org	Acc.Eqv	Inv.	$\bar{\kappa}$
GPT-4.1	93.1	89.8	87.9	85.4	0.75	70.9	83.8	77.7	0.33
GPT-4.1-mini	75.9	71.4	72.5	59.9	0.36	79.1	76.6	71.6	0.42
GPT-5	94.1	87.1	85.7	83.6	0.69	80.4	90.0	73.5	0.45
GPT-5-mini	70.0	72.0	67.1	54.6	0.26	70.5	71.4	66.0	0.26
Qwen3.5-27B	81.4	72.9	72.1	63.2	0.37	86.2	83.3	77.4	0.54

Table 1: Base ordering results pooled over

n\in\{4,8,\ldots,1024\}

(200 pairs per level). Acc.Org/Eqv score all prompts, counting invalid answers wrong; invariance (Inv.) and robust correctness (

R

) condition on jointly parsed answers.

\bar{\kappa}

averages chance-corrected invariance over the complexity grid. Percentages except

\bar{\kappa}

. Left/right renders the same latent instances with a different vocabulary.

Figure 2: Ordering competence profiles for the six base models on front/behind (200 pairs per chain length). Mean accuracy and both paired measures use the same valid-pair support. Robust correctness

R

requires both presentations to be correct; invariance

R+D

also includes both-wrong pairs. Shading gives pointwise 95% Wilson intervals for

R

. Accuracy can decline less than robust correctness, while shared errors can keep invariance high. Chance-corrected profiles for both vocabularies appear in Appendix E .

pooled over $n\in\{4,\dots,1024\}$
Effort	Acc.Org	Acc.Eqv	Inv.	$\bar{\kappa}$	tokens org/eqv
none	77.1	82.6	79.2	0.47	0/0
low	97.7	96.9	96.0	0.91	1,014/1,545
medium	99.4	98.8	98.4	0.97	1,641/2,718
high	99.9	99.4	99.4	0.99	2,052/3,455
$n=2048$ block (200 pairs; $\kappa_{2048}$ in the $\bar{\kappa}$ column)

Table 2: GPT-5.4 by reasoning effort on front/behind (200 pairs per

n

). Accuracy (%) on both presentations, paired invariance, chance correction, and mean reasoning tokens (original/equivalent). Every effort level uses the same prompts within each condition. The

n=2048

block uses a distinct presentation distribution (Appendix B ). No shown answer was truncated.

Panel of Figure 3 — Figure 3: GPT-5.4 reasoning-effort curves on front/behind (200 matched pairs per level). Mean accuracy and $\kappa_{n}$ share valid-pair support. Curves end at $n=1024$ ; isolated $n=2048$ markers, beyond the dashed divider, represent a separate presentation condition (Appendix B ). More reasoning extends strong performance, while the separate longer-chain condition retains failures.

Held-out front/behind
Arm	Acc.Org	Acc.Eqv	Inv.	$\bar{\kappa}$
9B base	55.6	53.3	57.3	0.00
SFT, cross-relation	56.4	55.0	56.6	$-$ 0.01
SFT, pairs	54.5	56.6	58.0	0.00
SFT, pairs + JS	56.2	54.5	58.3	0.00
GRPO	81.2	75.1	73.3	0.33

Table 3: Training comparisons (

n=4

1024

, 200 pairs per level). All adapters use LoRA on Qwen3.5-9B. GRPO reports seed 42, selected for higher mean invariance on the twelve clean held-out relations; the other seed is in Appendix G . JS adds a consistency loss. Accuracy uses all prompts; pair measures use jointly parsed answers. Percentages except

\bar{\kappa}

Panel of Figure 4 — Figure 4: Training changes the ordering complexity profile (held-out front/behind, 200 pairs per level). Mean accuracy and $\kappa_{n}$ share valid-pair support. The selected GRPO run is compared with the base 9B and SFT with a JS consistency loss. Gains at smaller sizes coexist with weak long-chain invariance.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Acc.Org	Acc.Eqv	Inv.	$\bar{\kappa}$	$R$	$D$	$U$
GPT-4o	60.2	61.7	75.0	0.46	48.5	26.6	25.0
GPT-4o-mini	55.2	53.9	79.2	0.46	44.2	35.1	20.8
GPT-4.1	62.5	60.7	79.8	0.41	51.5	28.3	20.2
GPT-4.1-mini	60.5	58.6	74.1	0.47	46.6	27.5	25.9
GPT-5	60.0	60.8	73.0	0.40	46.9	26.1	27.0
GPT-5-mini	57.0	58.3	72.2	0.37	43.8	28.4	27.8

Appendix

Table 4: Horn-clause inference pooled over

n\in\{4,\dots,128\}

rules (Protocol L, 200 pairs per

n

, pairs whose two answers both parse). Accuracy (%) on both presentations, per-pair invariance,

\bar{\kappa}

, and the joint readout

R

D

U

(both correct / both wrong / exactly one correct, %). No re-sampling floor was measured in this run.

Model		$n=4$	$n=8$	$n=16$	$n=32$	$n=64$	$n=128$
GPT-4o	$R$	82.5	61.5	40.8	34.9	34.0	36.4
	$D$	7.0	21.0	27.0	28.2	36.0	40.4
	$U$	10.5	17.5	32.1	36.9	29.9	23.2
	$\kappa_{n}$	0.78	0.60	0.31	0.26	0.34	0.48
GPT-4o-mini	$R$	55.5	50.5	40.5	38.5	35.0	45.0
	$D$	26.5	35.0	41.0	33.5	37.0	37.5

Appendix

Table 5: Horn-clause inference by number of rules

n

(Protocol L, 200 pairs per

n

R

D

U

: percentage of jointly parseable pairs with both presentations correct / both wrong / exactly one correct. Chance correction uses

\kappa_{n}=(\iota_{n}-\iota_{e})/(1-\iota_{e})

with

\iota_{n}=R+D

and

\iota_{e}=q^{2}+(1-q)^{2}

, where

q

is the pooled yes-rate on the same support. GPT-4o and Qwen3.5-9B lose a few pairs to unparseable answers. No retest control was collected.

Work	Generation and complexity	Presentations and diagnostic readout
CheckList ( Ribeiro et al., 2020 )	Templates, lexicons and perturbation tools construct behavioral tests.	Capability-specific tests include invariance and directional expectations, with failure rates.
Contrast sets ( Gardner et al., 2020 )	Experts minimally edit existing examples to probe local decision boundaries.	Edits typically change the gold label; contrast-set performance tests sensitivity to meaningful changes.
Metamorphic testing / LGMT ( Chen et al., 1998 ; Segura et al., 2016 ; Zhou et al., 2026 )	Follow-up cases are derived through specified metamorphic relations.	LGMT uses logical equivalences and cross-case consistency checks, with analyses by transformation type.
ParaRel / BECEL ( Elazar et al., 2021 ; Jang et al., 2022 )	Paraphrase patterns and task-specific transformations generate related inputs.	ParaRel includes correctness across all paraphrases; BECEL distinguishes consistency types and paired inconsistency.
ReClor / LogiQA ( Yu et al., 2020 ; Liu et al., 2020 )	Curated examination questions form fixed released datasets.	Accuracy analyses include question types or ReClor’s EASY/HARD subsets.
FOLIO / P-FOLIO ( Han et al., 2024a ; Han et al., 2024b )	Human-annotated arguments; P-FOLIO adds annotated reasoning chains.	Entailment, translation to logic, proof generation and inference-rule analyses provide distinct evaluation views.

Appendix

Table 6: Comparison of generation, complexity control, presentation changes and diagnostic readouts in the cited evaluation designs. Descriptions identify the kind of change being tested, including whether the answer or query changes. Further comparisons appear in Table 7 .

Work	Generation and complexity	Presentations and diagnostic readout
RobustLR ( Sanyal et al., 2022 )	Deductive inputs undergo minimal logical edits and equivalence transformations.	Logical-operator and equivalence-specific robustness analyses.
Premise Order Matters ( Chen et al., 2024 )	Deductive premises and GSM8K problems are reordered.	Answer-preserving reordering tests sensitivity to premise order.
The Reversal Curse ( Berglund et al., 2024 )	Synthetic-fact fine-tuning is evaluated using reverse-direction factual questions.	Tests generalization from a trained relation to reverse queries, which change the query direction.
The Illusion of Thinking ( Shojaee et al., 2025 )	Controllable puzzle environments vary problem complexity.	Final answers and reasoning traces are examined across complexity regimes.
LiveBench ( White et al., 2025 )	Regularly refreshed questions draw on recent sources and objective answers.	Task- and category-specific scores; freshness is distinct from a fixed-family complexity parameter.
Mirage ( Schaeffer et al., 2023 )	Reanalyzes model scaling and evaluation metrics.	Studies how metric choice changes the appearance of abrupt capability gains.

Appendix

Table 7: Further comparison of the evaluation designs described in the cited sources. Generating fresh inputs, varying problem complexity, testing presentation changes and reporting diagnostic measurements have multiple precedents. CBE connects these elements through its specified competence condition and matched profiles.

Model	Acc. Org	Acc. Eqv	Inv.	Inv. strict	Inv. $-$ floor	$\bar{\kappa}$
Qwen3.5-27B	82.9	77.1	74.5	74.5	$-6.7$	0.43
Qwen3.5-9B	66.2	65.4	62.7	62.4	$-4.7$	0.18
Gemma-4-31B	86.1	77.5	91.6	72.9	$-7.8$	0.82
Qwen3.6-35B-A3B	67.0	66.5	63.3	63.1	$-4.1$	0.18
Qwen2.5-32B	86.5	83.6	83.5	83.5	$-12.9$	0.59
9B + GRPO	89.4	87.6	87.1	87.1	$-8.8$	0.68

Appendix

Table 8: Means over the matched nine-relation core (Protocol P). Inv. strict counts a pair with an unparseable answer as a disagreement; Gemma’s gap between the two columns is format abandonment under the 20-token cap. The last row is the correctness-only GRPO adapter of § 6 .

Equivalent presentation	GPT-4.1	GPT-5
identical prompt re-sent (floor)	93.4 / 96.4	95.0 / 93.7
premise order reshuffled only	93.2 / 92.6	93.6 / 91.1
25% of premises re-worded	92.6 / 91.3	89.3 / 87.8
50% re-worded	90.3 / 89.9	84.8 / 85.5
75% re-worded	90.5 / 90.1	86.1 / 86.1
100% re-worded	84.1 / 80.2	82.4 / 81.4

Appendix

Table 9: What the presentation change is made of: Acc.Eqv / Inv. (%) on the front/behind instances of Protocol P when the equivalent presentation changes only the premise order, only the fraction of premises re-worded with the converse relation (order fixed), or both. Original-presentation accuracy is 93.1 (GPT-4.1) and 94.1 (GPT-5) in every row. The first row re-sends the original prompt unchanged.

	distractor dose $\rho$			presentation
Model	0	0.5	1	negation	retest
GPT-4.1	93.2	91.1	89.6	–	–
GPT-4.1-mini	76.0	76.2	74.3	–	–
GPT-5	92.9	91.0	89.8	90.0	97.0
GPT-5-mini	68.5	66.0	65.8	64.1	78.9
Qwen3.5-27B	75.2	72.7	69.6	60.2	81.8

Appendix

Table 10: Per-pair invariance (%) pooled over

n\in\{4,\dots,256\}

under Protocol L (see Section 4 ). Left: the syntactic equivalent with

\mathrm{round}(\rho\,(n-1))

distractor premises over disjoint entities added to both presentations. Right: the same items and premise subsets with each re-worded premise expressed as the negation of its converse, and the agreement of the original prompt with itself re-sent (retest). The

\rho=0

column is the syntactic family.

Relation	Acc.	Yes	$\delta$	$s$	Layer
Qwen3.5-9B, base
front/behind	55.5/59.5	96/92	1.58/1.14	0.63/0.50	16/18
left/right	69.5/68.5	82/84	1.67/1.41	1.65/1.38	13/13
nonce dir.	56.0/55.5	96/88	1.19/0.72	0.57/0.14	17/–
Qwen3.5-9B, correctness-only GRPO
front/behind	87.5/87.0	64/58	1.37/0.36	3.60/3.22	13/17

Appendix

Table 11: Output decomposition at the answer position (Protocol P instances,

n\leq 32

, 200 pairs per relation, greedy readout). Each cell is original/equivalent presentation. Acc. and yes-rate (Yes) in %;

\delta

= presentation-level default toward “yes” and

s

= instance signal, in logits; Layer = first layer at which the cross-validated gold probe reaches AUROC 0.9 (– = never; the Qwen3-8B replicate has 36 layers, the 9B 32). Nonce dir. = nonce direction words.

Figure 5: Gold-probe AUROC by layer at the answer position, Qwen3.5-9B (Protocol P instances, greedy readout,

n\leq 32

, 200 pairs per relation): base and correctness-only GRPO on front/behind, original (solid) and equivalent (dashed), and the equivalent presentation of the nonce direction words. Grey band: shuffled-label null; dotted line: AUROC 0.9.

Figure 6: GPT-5.4 by reasoning effort: per-pair invariance (top) and original-presentation accuracy (bottom) by

n

under front/behind (left) and left/right (right). Dotted: identical-prompt self-agreement; squares: the

n=2048

block. The none curves use the Protocol P instances with the Protocol L equivalents; low , medium , high and the

n=2048

block are Protocol L equivalents on the same instances.

Figure 7: Chance-corrected invariance

\kappa_{n}

by chain length for the six base models under front/behind (left) and left/right (right), Protocol P. Solid lines denote the larger and dashed lines the smaller model of each family. The dotted reference is

\kappa_{n}=0

under the pooled output-bias convention.

Model		4	8	16	32	64	128	256	512	1024
GPT-4.1-mini	Acc. Org	95.5	87.5	91.0	84.0	75.5	69.0	61.5	59.5	59.5
	Acc. Eqv	90.5	76.0	78.0	81.5	71.0	71.5	61.0	51.0	62.0
	Inv.	87.0	72.5	74.0	79.5	72.5	69.5	65.5	64.5	67.5
GPT-4.1	Acc. Org	98.5	100.0	99.5	99.5	97.0	94.0	89.5	82.0	77.5
	Acc. Eqv	98.0	98.5	100.0	96.0	95.0	92.5	86.0	74.0	68.5
	Inv.	97.5	98.5	99.5	95.5	92.0	87.9	81.5	71.0	67.3

Appendix

Table 12: Front/behind under Protocol P, per

n

(200 pairs each): accuracy on the original and equivalent presentations and per-pair invariance (%).

Model	Acc. Org	Acc. Eqv	$\Delta$	Inv.
GPT-4o-mini	71.7	67.5	+4.2	81.5
GPT-4o	83.2	82.3	+0.8	77.2
GPT-4.1-mini	71.6	74.1	-2.5	71.4
GPT-4.1	87.4	90.8	-3.4	84.4
GPT-5-mini	66.6	74.1	-7.5	67.3
GPT-5	90.6	88.3	+2.2	85.7

Appendix

Table 13: Per-model accuracy and invariance on the pos dataset (

N\!\in\!\{4,8,\ldots,2048\}

, 4000 questions per split). Acc. Org and Acc. Eqv are accuracy on original and equivalent splits;

\Delta

is their difference; Inv. is the fraction of paired questions where the model gave the same yes/no on both phrasings (over pairs with parseable answers). All values are percentages.

Model	$b$	$c$	Gap	95% CI	$\chi^{2}$	$p_{\text{McNemar}}$	$p_{\text{Holm}}$
GPT-4o-mini	453	285	+4.2	[+2.9, +5.5]	37.8	< 0.001	< 0.001
GPT-4o	473	439	+0.9	[-0.7, +2.3]	1.2	0.27	0.27
GPT-4.1-mini	522	621	-2.5	[-4.1, -0.9]	8.4	0.0037	0.0075
GPT-4.1	247	384	-3.4	[-4.7, -2.2]	29.3	< 0.001	< 0.001
GPT-5-mini	505	803	-7.4	[-9.3, -5.7]	67.4	< 0.001	< 0.001
GPT-5	332	243	+2.2	[+1.0, +3.4]	13.5	< 0.001	< 0.001

Appendix

Table 14: McNemar’s test on the original/equivalent accuracy gap, per model on the pos dataset (

N=4000

paired questions).

b

is the number of pairs with org correct and eqv wrong;

c

is the reverse. Gap = (org_acc

-

eqv_acc), with 95% paired-bootstrap CI (

B=10000

p_{\text{McNemar}}

uses the continuity-corrected

\chi^{2}

statistic;

p_{\text{Holm}}

is the Holm-Bonferroni adjusted

p

across the 8 models.

Model	$N$
	4	8	16	32	64	128	256	512	1024	2048
GPT-4o-mini	94.2	83.2	79.5	72.2	67.2	74.2	85.5	85.2	83.5	90.5
GPT-4o	88.5	84.5	89.8	88.5	85.0	80.0	75.0	66.0	59.5	55.2
GPT-4.1-mini	89.2	73.0	75.0	79.5	73.8	72.2	68.2	65.8	58.2	59.2
GPT-4.1	97.0	98.2	98.0	98.5	92.8	87.5	78.2	68.0	64.6	60.4
GPT-5-mini	86.8	68.2	71.2	73.8	64.5	60.2	56.2	57.8	62.0	72.2

Appendix

Table 15: Per-model invariance on the pos dataset, by problem size

N

. Invariance is the fraction of (original, equivalent) pairs receiving the same yes/no answer (over pairs with parseable answers). Values are percentages.

Figure 8: Per-pair invariance by

n

for the eight models under Protocol L (front/behind, 400 pairs per

n

). Dotted line: chance agreement.

Model	Acc. Org	Acc. Eqv	Mean	$\|\Delta\|$	Inv.
Qwen3.5-9B (base)	56.9	63.3	60.1	6.5	64.6
Qwen3.5-9B + LoRA	57.2	57.9	57.6	0.6	57.2
Qwen3.5-27B (base)	77.3	72.1	74.7	5.2	74.0
Qwen3.5-27B + LoRA	77.5	71.9	74.7	5.6	73.2

Appendix

Table 16: Cross-relation LoRA fine-tunes on the held-out pos relation under Protocol L, pooled over

n\in\{4,\ldots,2048\}

. Values recomputed from the released outputs.

Model	Names list	Acc. Org	Acc. Eqv	Inv.
Qwen3.5-9B (base)	shuffled	56.9	63.3	64.6
Qwen3.5-9B + LoRA	chain order	100.0	99.9	99.9
Qwen3.5-9B + LoRA	shuffled	62.3	66.1	78.1
Qwen3.5-27B	shuffled	77.3	72.1	74.0

Appendix

Table 17: A fine-tune trained on chain-ordered names lists, evaluated with chain-ordered and with shuffled names lists (Protocol L, pos , pooled over

n

). The

78.1

invariance under shuffled names coincides with a yes-default at large

n

(accuracy on no-instances about

18\%

n=2048

Model	Acc. Org	Acc. Eqv	$\Delta$	Inv.
GPT-4o-mini	94.6	94.7	$-0.1$	91.3
GPT-4o	94.6	93.4	$+1.2$	89.5
GPT-4.1-mini	87.9	92.6	$-4.7$	83.9
GPT-4.1	98.8	98.9	$-0.1$	97.6
GPT-5-mini	86.5	87.0	$-0.5$	78.2
GPT-5	96.4	93.8	$+2.7$	90.6

Appendix

Table 18: depth replication under Protocol L (

n\in\{4,\ldots,64\}

Held-out front/behind
Run	Acc.Org	Acc.Eqv	Inv.	$\bar{\kappa}$
Seed 42	81.2	75.1	73.3	0.33
Seed 43	78.1	76.7	67.3	0.32
Means over twelve clean held-out relations
Seed 42	87.6	83.7	83.5	0.60
Seed 43	87.9	85.6	81.3	0.61

Appendix

Table 19: GRPO training-seed replication (Protocol P,

n=4

1024

, 200 pairs per relation and level). Both runs use the same recipe. Main-text results select seed 42 by mean invariance across the twelve clean held-out relations. Accuracy uses all prompts; paired metrics use jointly parsed answers. Percentages except

\bar{\kappa}

Panel of Figure 9 — Figure 9: Training-seed replication on held-out front/behind (Protocol P, 200 pairs per chain length). Both GRPO seeds are compared with the base 9B and SFT with a JS consistency loss. Mean accuracy and $\kappa_{n}$ share valid-pair support. The main text displays seed 42, selected by mean invariance across the twelve clean held-out relations.

	Acc.				Inv.
Model	Org	Eqv	Inv.	$\bar{\kappa}$	$-$ floor
Qwen3.5-9B, base	61.4	60.2	60.0	0.10	$-$ 5.3
GRPO, seed 42	87.6	83.7	83.5	0.60	$-$ 12.4
GRPO, seed 43	87.9	85.6	81.3	0.61	$-$ 14.2
grpo-ig, $\lambda=0$ , seed 42	87.7	84.0	82.8	0.61	$-$ 12.1
grpo-ig, $\lambda=0.5$ , seed 42	88.8	84.6	84.6	0.63	$-$ 10.7

Appendix

Table 20: Agreement bonus versus correctness-only GRPO on the twelve clean held-out relations (Protocol P,

n=4

1024

, 1,800 pairs per relation). Acc., Inv. and

\bar{\kappa}

are means over the twelve relations; Inv.

-

floor over the eight relations with an identical-prompt floor. GRPO = per-presentation advantage groups (correctness-only in effect); grpo-ig = all sixteen samples of an instance in one advantage group, reward correct

+\lambda(\text{agree}-\text{chance})

Model	Inv.	Repeat	Inv. $-$ Repeat
GPT-4.1	87.9	96.4	$-8.5$
GPT-4.1-mini	72.5	83.6	$-11.1$
GPT-5	85.7	93.7	$-7.9$
GPT-5-mini	67.1	75.1	$-8.0$
Qwen3.5-27B	72.1	78.0	$-5.9$
Qwen3.5-9B	57.3	64.8	$-7.5$

Appendix

Table 21: Base-model repeat controls on front/behind (Protocol P,

n=4

1024

, 200 pairs per level). Inv. compares original and equivalent prompts; Repeat compares two responses to the identical original prompt. Values are percentages; differences are computed before rounding. Agreement conditions on parsed answers in each compared pair.

Arm	Inv.	Repeat	Inv. $-$ Repeat
9B base	57.3	64.8	$-7.5$
SFT, cross-relation	56.6	–	–
SFT, pairs	58.0	65.1	$-7.1$
SFT, pairs + JS	58.3	64.2	$-5.8$
GRPO, s42	73.3	93.7	$-20.4$
GRPO, s43	67.3	89.2	$-21.9$

Appendix

Table 22: Repeat controls for the Qwen3.5-9B training arms on held-out front/behind (Protocol P,

n=4

1024

, 200 pairs per level). Values are percentages; differences precede rounding. Dashes indicate an unavailable repeat control. Both SFT and GRPO use LoRA.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

May 12, 2026

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations rely on static benchmarks, which fail to assess robustness under logically equivalent transformations and often overestimate reasoning capability. We propose LGMT (Logic-Grounded Metamorphic Testing), an oracle-free framework that leverages first-order logic (FOL) to evaluate LLM reasoning. By deriving metamorphic relations from formal logical equivalences, LGMT constructs semantically invariant test cases and detects reasoning defects through cross-case consistency checking. Experiments on six state-of-the-art LLMs show that LGMT exposes substantial hidden defects missed by traditional reference-based evaluations. We further find that models are particularly sensitive to symbol-level and conclusion-level variations, and that advanced prompting such as Few-shot CoT only partially mitigates these issues. These results suggest that LLM evaluation should move beyond isolated correctness toward robustness under logical invariance. LGMT provides a principled and scalable approach for diagnosing reasoning failures.

Zenghui Zhou, Man Li, Xiaoke Fang +3

School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China

LLM Evaluation Automated Software Testing Metamorphic Testing LLM Reliability Language Model Robustness LLM Reasoning

Ask PDF

May 19, 2026

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

Evaluating large language models (LLMs) on natural-language logical reasoning is essential because rule-governed tasks require conclusions to follow strictly from stated premises. Many existing logical-reasoning benchmarks are generated by templating natural-language items from sampled formulas, provide only coarse or unaudited formal annotations, and are now quickly saturated by frontier reasoning models. We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios. Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow. The benchmark is released in two paired subsets: a 246-item Base subset shipped with 1,400 expert-developed rubric atoms, and a 190-item Hard subset with 938 multi-step sub-questions over closed model spaces. Evaluating 14 frontier LLMs on LLMEval-Logic reveals substantial gaps in current models: the best model reaches only 37.5% Hard Item Accuracy, and even with reference symbols the highest joint Z3+Rubric formalization score among evaluated models reaches only 60.16%. Our benchmark is publicly available at https://github.com/llmeval/LLMEval-Logic.

Ming Zhang, Qiyuan Peng, Yinxi Wei +13

Institute of Trustworthy Embodied Artificial Intelligence, Fudan University · 2Hunyuan Team, Tencent · School of Philosophy, Fudan University

Logical Reasoning LLM Evaluation LLM Reasoning

Ask PDF

Mar 16, 2026

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

Yuzhe Tang

Georgia Institute of Technology

Language Model Reasoning Evaluation LLM Evaluation Implicit Reasoning in Language Models Reasoning Consistency in Language Models

Ask PDF

Abstract

LLM Evaluation

Explore similar work

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models