ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

İbrahim Ethem Deveci; Funda Tan Çalık; Barış Deniz Sağlam; Duygu Ataman

cs.AISep 29, 2026

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

Authors: İbrahim Ethem Deveci, Funda Tan Çalık, Barış Deniz Sağlam, Duygu Ataman

Organizations: Graduate School of Informatics Middle East Technical University Ankara, Türkiye

Abstract

Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic. These settings make model accuracy easier to evaluate and optimize, but it remains unclear how far success under fixed problem specifications and stable evaluation criteria transfers to reasoning outside such domains. Real-world reasoning often proceeds under incomplete and revisable information: conclusions may be supported provisionally, defeated by counter-evidence, reinstated by further arguments, or revised when stronger reasons become available. Reasoning of this kind is generally referred to as defeasible reasoning. We introduce ArgGYM, a procedural benchmark and RLVR-compatible training environment for structured defeasible reasoning. ArgGYM decomposes this reasoning into twelve tasks and grounds task-specific scoring in a symbolic argumentation engine that computes the formal states used to evaluate model outputs. It includes a frozen benchmark of 1,440 verified instances across fifteen curriculum configurations, two argument preference orderings (weakest-link and last-link), and two set orderings (elitist and democratic), while the same generators and verifiers can produce fresh instances for evaluation that reduces dependence on static test sets and for verifiable-reward training. On the frozen benchmark, frontier and open-weight models show sharply different reasoning profiles: they can recover substantial parts of structured answers without solving the complete task, and performance declines in later curriculum configurations with longer dependencies and more interacting structures. We release the benchmark, generators, and verifiers for reproducible evaluation and RLVR training.

Figures & tables

Task	Objective
Representation and support
formalization	Construct an executable ASPIC+ theory from a controlled natural-language description whose induced argumentative state is behaviorally equivalent to the intended theory.
claim_chain	Recover the complete support derivation for a justified target claim in an ordering consistent with its derivational dependencies.
Diagnosis, semantics, and revision
defeat_diagnosis	Identify where support for a non-justified target fails and the relevant defeaters responsible for that failure.
status_query	Determine the statuses of queried claims under grounded semantics.

Table 1: The twelve ArgGYM tasks. The first six require structured analysis or formalization outputs, while the final six require constructive interventions whose consequences are executed and verified through the task-specific scoring pipeline.

Model	Complete-task success (%)
Gemma
Gemma 4 E2B	off 1.1 $\rightarrow$ on 1.3
Gemma 4 E4B	off 3.5 $\rightarrow$ on 3.8
Gemma 4 26B-A4B	off 9.0 $\rightarrow$ on 14.1
Gemma 4 31B	off 14.4 $\rightarrow$ on 27.3
GPT-OSS

Table 2: Overall ArgGYM performance. Complete-task success (%) across reasoning configurations.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Reasoning	Sampling	Seed	Max. out.	Context
Gemma
google/gemma-4-E2B-it	off / on	$T=1.0,\ p=.95,\ k=64$	0	65536	131072
google/gemma-4-E4B-it	off / on	$T=1.0,\ p=.95,\ k=64$	0	65536	131072
google/gemma-4-26B-A4B-it	off / on	$T=1.0,\ p=.95,\ k=64$	0	65536	262144
google/gemma-4-31B-it	off / on	$T=1.0,\ p=.95,\ k=64$	0	65536	262144
GPT-OSS

Appendix

Table 3: Model and inference configurations.

T

denotes temperature,

p

top-

p

k

top-

k

, and PP presence penalty. Max. out. is the maximum generated-token budget.

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.8 $\|$ 1.0	2.5 $\|$ 3.5	5.0 $\|$ 13.1	1.7 $\|$ 47.6	0.0 $\|$ 38.0	0.0 $\|$ 3.7
	on	0.8 $\|$ 1.5	5.0 $\|$ 5.3	0.0 $\|$ 10.0	4.2 $\|$ 52.1	0.0 $\|$ 42.2	0.0 $\|$ 4.8
google/gemma-4-E4B-it	off	4.2 $\|$ 16.8	2.5 $\|$ 3.3	5.8 $\|$ 23.5	11.7 $\|$ 69.4	2.5 $\|$ 58.6	0.8 $\|$ 12.7
	on	4.2 $\|$ 25.1	6.7 $\|$ 7.9	9.2 $\|$ 26.1	10.8 $\|$ 71.4	0.0 $\|$ 56.8	0.8 $\|$ 16.2
google/gemma-4-26B-A4B-it	off	7.5 $\|$ 43.2	2.5 $\|$ 3.3	9.2 $\|$ 35.5	15.0 $\|$ 75.1	4.2 $\|$ 64.5	1.7 $\|$ 22.8

Appendix

Table 4: Analysis and evaluation tasks. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	2.5 $\|$ 2.7	0.0 $\|$ 0.0	0.0 $\|$ 0.1	0.0 $\|$ 0.9	0.8 $\|$ 0.8	0.0 $\|$ 0.6
	on	3.3 $\|$ 3.5	0.0 $\|$ 0.2	0.0 $\|$ 0.2	0.0 $\|$ 1.2	2.5 $\|$ 2.5	0.0 $\|$ 1.0
google/gemma-4-E4B-it	off	12.5 $\|$ 14.7	0.0 $\|$ 2.1	0.0 $\|$ 1.8	0.0 $\|$ 3.9	2.5 $\|$ 4.5	0.0 $\|$ 2.4
	on	8.3 $\|$ 11.6	0.0 $\|$ 2.3	0.0 $\|$ 3.6	0.0 $\|$ 4.2	4.2 $\|$ 6.8	0.8 $\|$ 5.0
google/gemma-4-26B-A4B-it	off	30.8 $\|$ 33.4	2.5 $\|$ 4.9	4.2 $\|$ 6.8	0.0 $\|$ 4.3	20.8 $\|$ 22.1	10.0 $\|$ 15.2

Appendix

Table 5: Constructive intervention tasks. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	L1–L5	L6–L10	L11–L15
google/gemma-4-E2B-it	off	3.3	0.0	0.0
	on	4.0	0.0	0.0
google/gemma-4-E4B-it	off	9.6	0.8	0.2
	on	10.2	0.6	0.4
google/gemma-4-26B-A4B-it	off	17.5	5.6	4.0

Appendix

Table 6: Complete-task success across structural-richness bands. Values are percentages.

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	2.5 $\|$ 3.1	7.5 $\|$ 8.4	15.0 $\|$ 26.2	5.0 $\|$ 56.0	0.0 $\|$ 40.2	0.0 $\|$ 8.2
	on	2.5 $\|$ 4.6	15.0 $\|$ 15.7	0.0 $\|$ 16.1	12.5 $\|$ 65.2	0.0 $\|$ 47.3	0.0 $\|$ 9.1
google/gemma-4-E4B-it	off	12.5 $\|$ 32.0	5.0 $\|$ 5.0	17.5 $\|$ 50.7	35.0 $\|$ 81.5	7.5 $\|$ 71.0	2.5 $\|$ 28.0
	on	12.5 $\|$ 35.3	15.0 $\|$ 16.7	27.5 $\|$ 54.4	32.5 $\|$ 84.8	0.0 $\|$ 62.1	2.5 $\|$ 30.7
google/gemma-4-26B-A4B-it	off	20.0 $\|$ 49.0	7.5 $\|$ 7.5	27.5 $\|$ 73.8	42.5 $\|$ 86.9	12.5 $\|$ 75.2	5.0 $\|$ 37.5

Appendix

Table 7: Task-specific analysis/evaluation performance for curriculum configurations L1–L5. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	7.5 $\|$ 8.2	0.0 $\|$ 0.1	0.0 $\|$ 0.2	0.0 $\|$ 2.2	2.5 $\|$ 2.5	0.0 $\|$ 1.4
	on	10.0 $\|$ 10.0	0.0 $\|$ 0.4	0.0 $\|$ 0.6	0.0 $\|$ 3.1	7.5 $\|$ 7.5	0.0 $\|$ 1.9
google/gemma-4-E4B-it	off	30.0 $\|$ 31.8	0.0 $\|$ 4.8	0.0 $\|$ 4.4	0.0 $\|$ 7.0	5.0 $\|$ 5.0	0.0 $\|$ 3.9
	on	20.0 $\|$ 21.4	0.0 $\|$ 3.6	0.0 $\|$ 5.8	0.0 $\|$ 6.9	12.5 $\|$ 12.5	0.0 $\|$ 5.2
google/gemma-4-26B-A4B-it	off	42.5 $\|$ 43.2	7.5 $\|$ 10.2	5.0 $\|$ 8.0	0.0 $\|$ 8.5	32.5 $\|$ 32.5	7.5 $\|$ 13.9

Appendix

Table 8: Task-specific constructive performance for curriculum configurations L1–L5. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.0 $\|$ 0.0	0.0 $\|$ 1.8	0.0 $\|$ 6.4	0.0 $\|$ 44.6	0.0 $\|$ 36.2	0.0 $\|$ 2.6
	on	0.0 $\|$ 0.0	0.0 $\|$ 0.2	0.0 $\|$ 7.1	0.0 $\|$ 47.8	0.0 $\|$ 39.9	0.0 $\|$ 2.7
google/gemma-4-E4B-it	off	0.0 $\|$ 2.4	2.5 $\|$ 4.0	0.0 $\|$ 11.7	0.0 $\|$ 67.8	0.0 $\|$ 54.0	0.0 $\|$ 6.9
	on	0.0 $\|$ 15.2	2.5 $\|$ 3.8	0.0 $\|$ 13.2	0.0 $\|$ 70.7	0.0 $\|$ 55.6	0.0 $\|$ 10.5
google/gemma-4-26B-A4B-it	off	2.5 $\|$ 40.8	0.0 $\|$ 0.0	0.0 $\|$ 22.2	2.5 $\|$ 72.2	0.0 $\|$ 57.1	0.0 $\|$ 20.6

Appendix

Table 9: Task-specific analysis/evaluation performance for curriculum configurations L6–L10. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.4	0.0 $\|$ 0.0	0.0 $\|$ 0.4
	on	0.0 $\|$ 0.4	0.0 $\|$ 0.2	0.0 $\|$ 0.0	0.0 $\|$ 0.4	0.0 $\|$ 0.0	0.0 $\|$ 0.6
google/gemma-4-E4B-it	off	5.0 $\|$ 6.8	0.0 $\|$ 1.4	0.0 $\|$ 0.9	0.0 $\|$ 2.6	2.5 $\|$ 4.5	0.0 $\|$ 2.1
	on	2.5 $\|$ 7.4	0.0 $\|$ 2.6	0.0 $\|$ 2.5	0.0 $\|$ 3.3	0.0 $\|$ 1.7	2.5 $\|$ 6.2
google/gemma-4-26B-A4B-it	off	27.5 $\|$ 30.9	0.0 $\|$ 2.1	2.5 $\|$ 5.8	0.0 $\|$ 3.6	20.0 $\|$ 20.7	12.5 $\|$ 17.9

Appendix

Table 10: Task-specific constructive performance for curriculum configurations L6–L10. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.0 $\|$ 0.0	0.0 $\|$ 0.3	0.0 $\|$ 6.7	0.0 $\|$ 42.1	0.0 $\|$ 37.7	0.0 $\|$ 0.5
	on	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 6.7	0.0 $\|$ 43.4	0.0 $\|$ 39.5	0.0 $\|$ 2.5
google/gemma-4-E4B-it	off	0.0 $\|$ 15.9	0.0 $\|$ 0.8	0.0 $\|$ 8.2	0.0 $\|$ 59.0	0.0 $\|$ 50.7	0.0 $\|$ 3.3
	on	0.0 $\|$ 24.9	2.5 $\|$ 3.3	0.0 $\|$ 10.7	0.0 $\|$ 58.6	0.0 $\|$ 52.8	0.0 $\|$ 7.5
google/gemma-4-26B-A4B-it	off	0.0 $\|$ 39.8	0.0 $\|$ 2.3	0.0 $\|$ 10.6	0.0 $\|$ 66.2	0.0 $\|$ 61.2	0.0 $\|$ 10.2

Appendix

Table 11: Task-specific analysis/evaluation performance for curriculum configurations L11–L15. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.1	0.0 $\|$ 0.0	0.0 $\|$ 0.1
	on	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.1	0.0 $\|$ 0.0	0.0 $\|$ 0.4
google/gemma-4-E4B-it	off	2.5 $\|$ 5.5	0.0 $\|$ 0.2	0.0 $\|$ 0.0	0.0 $\|$ 2.2	0.0 $\|$ 4.0	0.0 $\|$ 1.1
	on	2.5 $\|$ 6.0	0.0 $\|$ 0.8	0.0 $\|$ 2.4	0.0 $\|$ 2.3	0.0 $\|$ 6.3	0.0 $\|$ 3.5
google/gemma-4-26B-A4B-it	off	22.5 $\|$ 26.2	0.0 $\|$ 2.2	5.0 $\|$ 6.7	0.0 $\|$ 0.8	10.0 $\|$ 13.2	10.0 $\|$ 13.8

Appendix

Table 12: Task-specific constructive performance for curriculum configurations L11–L15. Each cell reports complete-task success (%)

\,|\,

task-native score

\times 100

Gemma
Model	Reasoning	Last-link	Weakest-link	Democratic	Elitist
google/gemma-4-E2B-it	off	1.0	1.2	1.4	0.8
	on	1.5	1.1	1.2	1.4
google/gemma-4-E4B-it	off	5.1	1.9	3.9	3.2
	on	4.9	2.6	3.9	3.6
google/gemma-4-26B-A4B-it	off	14.0	4.0	9.3	8.8

Appendix

Table 13: Performance by argument and set ordering. Values are complete-task success (%). Last-link and weakest-link refer to the argument comparison principle; democratic and elitist refer to the set ordering used for preference comparison.

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.0 $\|$ 1.7	0.0 $\|$ 5.0	5.0 $\|$ 5.0	0.0 $\|$ 3.3	0.0 $\|$ 0.0	0.0 $\|$ 0.0
	on	0.0 $\|$ 1.7	3.3 $\|$ 6.7	0.0 $\|$ 0.0	3.3 $\|$ 5.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0
google/gemma-4-E4B-it	off	5.0 $\|$ 3.3	3.3 $\|$ 1.7	6.7 $\|$ 5.0	13.3 $\|$ 10.0	3.3 $\|$ 1.7	0.0 $\|$ 1.7
	on	5.0 $\|$ 3.3	5.0 $\|$ 8.3	10.0 $\|$ 8.3	11.7 $\|$ 10.0	0.0 $\|$ 0.0	1.7 $\|$ 0.0
google/gemma-4-26B-A4B-it	off	10.0 $\|$ 5.0	3.3 $\|$ 1.7	10.0 $\|$ 8.3	15.0 $\|$ 15.0	3.3 $\|$ 5.0	3.3 $\|$ 0.0

Appendix

Table 14: Task-specific last-link versus weakest-link performance on analysis/evaluation tasks. Each cell reports complete-task success as last-link (%)

|

weakest-link (%).

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	5.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	1.7 $\|$ 0.0	0.0 $\|$ 0.0
	on	6.7 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	5.0 $\|$ 0.0	0.0 $\|$ 0.0
google/gemma-4-E4B-it	off	25.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	5.0 $\|$ 0.0	0.0 $\|$ 0.0
	on	16.7 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	0.0 $\|$ 0.0	8.3 $\|$ 0.0	0.0 $\|$ 1.7
google/gemma-4-26B-A4B-it	off	61.7 $\|$ 0.0	3.3 $\|$ 1.7	3.3 $\|$ 5.0	0.0 $\|$ 0.0	35.0 $\|$ 6.7	20.0 $\|$ 0.0

Appendix

Table 15: Task-specific last-link versus weakest-link performance on constructive tasks. Each cell reports complete-task success as last-link (%)

|

weakest-link (%).

Gemma
Model	Reasoning	Trunc.	No answer	Compliance	Canonical	Cond.	$\Delta$
google/gemma-4-E2B-it	off	0.4	0.8	99.2	1.1	1.1	+0.0
	on	0.1	20.9	79.1	1.3	1.7	+0.3
google/gemma-4-E4B-it	off	0.0	0.2	99.8	3.5	3.5	+0.0
	on	0.1	4.2	95.8	3.8	3.9	+0.2
google/gemma-4-26B-A4B-it	off	10.3	9.9	90.1	9.0	10.0	+1.0

Appendix

Table 16: Generation and answer-protocol diagnostics. Trunc. is the percentage of truncated generations; No answer is the percentage without the required <answer> region; Compliance is the corresponding answer-region compliance rate. Cond. denotes success conditioned on the presence of an answer region, and

\Delta

is conditioned minus canonical success in percentage points.

Explore similar work

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark
Aug 31, 2026Jayanta Sadhu, Sayem Shahad, Kenneth MarinoConversational Context
University of Utah · Bangladesh University of Engineering and Technology
LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
May 19, 2026Ming Zhang, Qiyuan Peng, Yinxi Wei +13Reasoning Benchmark Reasoning Skills
Institute of Trustworthy Embodied Artificial Intelligence, Fudan University · 2Hunyuan Team, Tencent · School of Philosophy, Fudan University
DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
Jun 17, 2026Patrick Cooper, Alvaro VelasquezAbductive Reasoning Foundation Model
University of Colorado Boulder

Aug 31, 2026, cs.AIJ/K move · Enter open · S save

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark

Jayanta Sadhu, Sayem Shahad, Kenneth Marino

University of Utah · Bangladesh University of Engineering and Technology

Conversational Context

Open paper AskPDF arXiv

Defeasible reasoning is a type of reasoning where inferences are drawn from plausible current evidence, but can be retracted upon the introduction of newer evidence. Although recent studies have examined language-model behaviors in defeasible reasoning, the datasets have been static and lack wide coverage of non-monotonic reasoning categories. We introduce DeReLab, a generative framework that produces multi-turn belief-updating conversations from parameterized graph structures across default and inheritance reasoning, with formally verified ground truth at every turn, enabling controlled measurement of how models respond to confirming and disconfirming evidence. This controlled generation process creates a testbed for experimental designs that isolate specific reasoning demands. Applying this capability to the study of confirmation bias, we evaluate nine open and proprietary large language models and find that nearly all exhibit a systematic tendency to accept congruent evidence while resisting incongruent updates, with several models correctly identifying a weakening update yet failing to revise their conclusion. We believe our work and findings will facilitate future research on evaluating language models in defeasible reasoning.

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.8 $\|$ 1.0	2.5 $\|$ 3.5	5.0 $\|$ 13.1	1.7 $\|$ 47.6	0.0 $\|$ 38.0	0.0 $\|$ 3.7
	on	0.8 $\|$ 1.5	5.0 $\|$ 5.3	0.0 $\|$ 10.0	4.2 $\|$ 52.1	0.0 $\|$ 42.2	0.0 $\|$ 4.8
google/gemma-4-E4B-it	off	4.2 $\|$ 16.8	2.5 $\|$ 3.3	5.8 $\|$ 23.5	11.7 $\|$ 69.4	2.5 $\|$ 58.6	0.8 $\|$ 12.7
	on	4.2 $\|$ 25.1	6.7 $\|$ 7.9	9.2 $\|$ 26.1	10.8 $\|$ 71.4	0.0 $\|$ 56.8	0.8 $\|$ 16.2
google/gemma-4-26B-A4B-it	off	7.5 $\|$ 43.2	2.5 $\|$ 3.3	9.2 $\|$ 35.5	15.0 $\|$ 75.1	4.2 $\|$ 64.5	1.7 $\|$ 22.8

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	2.5 $\|$ 2.7	0.0 $\|$ 0.0	0.0 $\|$ 0.1	0.0 $\|$ 0.9	0.8 $\|$ 0.8	0.0 $\|$ 0.6
	on	3.3 $\|$ 3.5	0.0 $\|$ 0.2	0.0 $\|$ 0.2	0.0 $\|$ 1.2	2.5 $\|$ 2.5	0.0 $\|$ 1.0
google/gemma-4-E4B-it	off	12.5 $\|$ 14.7	0.0 $\|$ 2.1	0.0 $\|$ 1.8	0.0 $\|$ 3.9	2.5 $\|$ 4.5	0.0 $\|$ 2.4
	on	8.3 $\|$ 11.6	0.0 $\|$ 2.3	0.0 $\|$ 3.6	0.0 $\|$ 4.2	4.2 $\|$ 6.8	0.8 $\|$ 5.0
google/gemma-4-26B-A4B-it	off	30.8 $\|$ 33.4	2.5 $\|$ 4.9	4.2 $\|$ 6.8	0.0 $\|$ 4.3	20.8 $\|$ 22.1	10.0 $\|$ 15.2

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	2.5 $\|$ 3.1	7.5 $\|$ 8.4	15.0 $\|$ 26.2	5.0 $\|$ 56.0	0.0 $\|$ 40.2	0.0 $\|$ 8.2
	on	2.5 $\|$ 4.6	15.0 $\|$ 15.7	0.0 $\|$ 16.1	12.5 $\|$ 65.2	0.0 $\|$ 47.3	0.0 $\|$ 9.1
google/gemma-4-E4B-it	off	12.5 $\|$ 32.0	5.0 $\|$ 5.0	17.5 $\|$ 50.7	35.0 $\|$ 81.5	7.5 $\|$ 71.0	2.5 $\|$ 28.0
	on	12.5 $\|$ 35.3	15.0 $\|$ 16.7	27.5 $\|$ 54.4	32.5 $\|$ 84.8	0.0 $\|$ 62.1	2.5 $\|$ 30.7
google/gemma-4-26B-A4B-it	off	20.0 $\|$ 49.0	7.5 $\|$ 7.5	27.5 $\|$ 73.8	42.5 $\|$ 86.9	12.5 $\|$ 75.2	5.0 $\|$ 37.5

Gemma
Model	Reasoning	Pref.	Counter	Counter+S	Attack	Defence	Attack–Def.
google/gemma-4-E2B-it	off	7.5 $\|$ 8.2	0.0 $\|$ 0.1	0.0 $\|$ 0.2	0.0 $\|$ 2.2	2.5 $\|$ 2.5	0.0 $\|$ 1.4
	on	10.0 $\|$ 10.0	0.0 $\|$ 0.4	0.0 $\|$ 0.6	0.0 $\|$ 3.1	7.5 $\|$ 7.5	0.0 $\|$ 1.9
google/gemma-4-E4B-it	off	30.0 $\|$ 31.8	0.0 $\|$ 4.8	0.0 $\|$ 4.4	0.0 $\|$ 7.0	5.0 $\|$ 5.0	0.0 $\|$ 3.9
	on	20.0 $\|$ 21.4	0.0 $\|$ 3.6	0.0 $\|$ 5.8	0.0 $\|$ 6.9	12.5 $\|$ 12.5	0.0 $\|$ 5.2
google/gemma-4-26B-A4B-it	off	42.5 $\|$ 43.2	7.5 $\|$ 10.2	5.0 $\|$ 8.0	0.0 $\|$ 8.5	32.5 $\|$ 32.5	7.5 $\|$ 13.9

Gemma
Model	Reasoning	Formal.	Chain	Diagnosis	Status	Semantics	Perturb.
google/gemma-4-E2B-it	off	0.0 $\|$ 0.0	0.0 $\|$ 1.8	0.0 $\|$ 6.4	0.0 $\|$ 44.6	0.0 $\|$ 36.2	0.0 $\|$ 2.6
	on	0.0 $\|$ 0.0	0.0 $\|$ 0.2	0.0 $\|$ 7.1	0.0 $\|$ 47.8	0.0 $\|$ 39.9	0.0 $\|$ 2.7
google/gemma-4-E4B-it	off	0.0 $\|$ 2.4	2.5 $\|$ 4.0	0.0 $\|$ 11.7	0.0 $\|$ 67.8	0.0 $\|$ 54.0	0.0 $\|$ 6.9
	on	0.0 $\|$ 15.2	2.5 $\|$ 3.8	0.0 $\|$ 13.2	0.0 $\|$ 70.7	0.0 $\|$ 55.6	0.0 $\|$ 10.5
google/gemma-4-26B-A4B-it	off	2.5 $\|$ 40.8	0.0 $\|$ 0.0	0.0 $\|$ 22.2	2.5 $\|$ 72.2	0.0 $\|$ 57.1	0.0 $\|$ 20.6

Abstract

Explore similar work

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models