Evaluation of Deontic Conditional Reasoning in Large Language Models: The Case of Wason's Selection Task

Hirohiko Abe; Kentaro Ozeki; Risako Ando; Takanobu Morishita; Koji Mineshima; Mitsuhiro Okada

cs.CLMar 6, 2026

Evaluation of Deontic Conditional Reasoning in Large Language Models: The Case of Wason's Selection Task

Authors: Hirohiko Abe, Kentaro Ozeki, Risako Ando, Takanobu Morishita, Koji Mineshima, Mitsuhiro Okada

Organizations: Keio University, Tokyo, Japan · University of Tokyo, Tokyo, Japan

Abstract

As large language models (LLMs) advance in linguistic competence, their reasoning abilities are gaining increasing attention. In humans, reasoning often performs well in domain specific settings, particularly in normative rather than purely formal contexts. Although prior studies have compared LLM and human reasoning, the domain specificity of LLM reasoning remains underexplored. In this study, we introduce a new Wason Selection Task dataset that explicitly encodes deontic modality to systematically distinguish deontic from descriptive conditionals, and use it to examine LLMs' conditional reasoning under deontic rules. We further analyze whether observed error patterns are better explained by confirmation bias (a tendency to seek rule-supporting evidence) or by matching bias (a tendency to ignore negation and select items that lexically match elements of the rule). Results show that, like humans, LLMs reason better with deontic rules and display matching-bias-like errors. Together, these findings suggest that the performance of LLMs varies systematically across rule types and that their error patterns can parallel well-known human biases in this paradigm.

Figures & tables

Rule Polarity	Pattern	Example Rule	TA	FA	TC	FC
Pos-Pos	$p\to q$	If the number is prime, then the letter is lowercase.	7	8	b	B
Pos-Neg	$p\to\neg q$	If the number is prime, then the letter is not lowercase.	7	8	B	b
Neg-Pos	$\neg p\to q$	If the number is not prime, then the letter is lowercase.	8	7	b	B
Neg-Neg	$\neg p\to\neg q$	If the number is not prime, then the letter is not lowercase.	8	7	B	b

Table 1: List of patterns involving negation. Pos : affirmative, Neg : negated. TA : case where the antecedent is true (True Antecedent), FA : case where the antecedent is false (False Antecedent), TC : case where the consequent is true (True Consequent), FC : case where the consequent is false (False Consequent). Cells highlighted in blue indicate logically correct selections. indicates options predicted by matching bias. Confirmation bias predicts selections of both TA and TC .

Reasoning Models
Model	Zero	Few	CoT
gpt-oss-20b	67.5 ( ${\color[rgb]{0,0,1}+27.5}$ )	91.2 ( ${\color[rgb]{0,0,1}+41.2}$ )	n/a
gpt-oss-120b	100.0 ( ${\color[rgb]{0,0,1}+30}$ )	98.8 ( ${\color[rgb]{0,0,1}+30}$ )	n/a
qwen3-14b	97.5 ( ${\color[rgb]{0,0,1}+45}$ )	98.8 ( ${\color[rgb]{0,0,1}+37.5}$ )	n/a
qwen3-32b	98.8 ( ${\color[rgb]{0,0,1}+38.8}$ )	100.0 ( ${\color[rgb]{0,0,1}+25}$ )	n/a
Non-Reasoning Models

Table 2: Accuracy (%) by prompt type for deontic rule problems. Values in parentheses show the difference in accuracy from the descriptive rule condition (positive if deontic ¿ descriptive). Zero = Zero-Shot, Few = Few-Shot, CoT = Chain-of-Thought.

Polarity	Pattern	TA	FA	TC	FC
pos-pos	$p\to q$	100.0	0.0	45.0	55.0
pos-neg	$p\to\lnot q$	100.0	0.0	5.0	95.0
neg-pos	$\lnot p\to q$	80.0	20.0	60.0	40.0
neg-neg	$\lnot p\to\lnot q$	90.0	10.0	10.0	90.0

Table 3: Selection percentages (%) for each option in the deontic rule problems (gpt-oss-20b, Zero-Shot setting). As with Table 1 , cells highlighted in blue indicate logically correct selections. indicates options predicted by matching bias. Confirmation bias predicts selections of both TA and TC .

Table 4: Percentage (%) of types of options selected by the models, relative to rule polarity (Zero-Shot setting). For the meanings of TA, FA, TC, FC , refer to Section 3 and Table 1 . Boldface indicates the higher value between

p

and

\neg p

, or

q

and

\neg q

, in each respective column (when applicable).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Rule Polarity	Pattern	Descriptive	Deontic
Pos-Pos	$p\to q$	If a number is prime, then the opposite side has a lowercase letter.	If the lamp is on, then the visitor must enter the room.
Pos-Neg	$p\to\neg q$	If a number is prime, then the opposite side does not have a lowercase letter.	If the lamp is on, then the visitor must not enter the room.
Neg-Pos	$\neg p\to q$	If a number is not prime, then the opposite side has a lowercase letter.	If the lamp is not on, then the visitor must enter the room.
Neg-Neg	$\neg p\to\neg q$	If a number is not prime, then the opposite side does not have a lowercase letter.	If the lamp is not on, then the visitor must not enter the room.

Appendix

Table 5: Example rules of descriptive and deontic cases.

Panel of Figure 3 — Figure 3: An example of error for a descriptive rule with Pos-Pos polarity.

Table 6: gpt-oss-20b (Zero-Shot)

Table 7: gpt-oss-20b (Few-Shot)

Table 8: gpt-oss-120b (Zero-Shot)

Table 9: gpt-oss-120b (Few-Shot)

Table 10: qwen3-14b (Zero-Shot)

Table 11: qwen3-14b (Few-Shot)

Table 12: qwen3-32b (Zero-Shot)

Table 13: qwen3-32b (Few-Shot)

Table 14: gemma3-4b (Zero-Shot)

Table 15: gemma3-4b (Few-Shot)

Table 16: gemma3-4b (CoT)

Table 17: gemma3-12b (Zero-Shot)

Table 18: gemma3-12b (Few-Shot)

Table 19: gemma3-12b (CoT)

Table 20: gemma3-27b (Zero-Shot)

Table 21: gemma3-27b (Few-Shot)

Table 22: gemma3-27b (CoT)

Table 23: llama-3.3-70b (Zero-Shot)

Table 24: llama-3.3-70b (Few-Shot)

Table 25: llama-3.3-70b (CoT)

Table 26: olmo-2-0325-32b (Zero-Shot)

Table 27: olmo-2-0325-32b (Few-Shot)

Table 28: olmo-2-0325-32b (CoT)

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Tracing the ongoing emergence of human-like reasoning in Large Language Models
May 20, 2026Paolo Morosi, Nikoleta Pantelidou, Fritz Günther +2Logical Reasoning Pragmatic Reasoning in Language Models
Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain
Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
May 18, 2026Tara Azin, Yongan Yu, Raj Singh +1LLM Evaluation Pragmatic Reasoning in Language Models
Department of Cognitive Science, Carleton University · School of Computer Science, McGill University · Mila – Quebec AI Institute
Humans and LLMs Diverge on Probabilistic Inferences
Feb 26, 2026Gaurav Kamath, Sreenath Madathil, Sebastian Schuster +2LLM Reasoning
McGill University · Mila – Quebec AI Institute · University of Vienna +2

May 20, 2026J/K move · Enter open · S save

Tracing the ongoing emergence of human-like reasoning in Large Language Models

Paolo Morosi, Nikoleta Pantelidou, Fritz Günther, Elena Pagliarini+1

Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain

Logical Reasoning Pragmatic Reasoning in Language Models LLM Reasoning Natural Language Inference

Open paper AskPDF arXiv

Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will pay only if the lawn is mowed, whereas If you are hungry, there is pizza in the oven implies that pizza is available regardless of the hearers hunger. Large Language Models - LLMs - show human-like performance on many tasks, yet it remains unclear whether they reason like humans. To address this, we conducted a population-matching experiment assessing how twentyfive LLMs compute conditional inferences across four languages, compared to an equal number of humans per language. We find that humans enrich logical reasoning through pragmatic inferences across languages. Model behavior is more variable. Some LLMs perfectly follow the truth-table of conditionals but they ignore pragmatic inferences, while others deviate from the truth-table, adhering to a single interpretation across the board, thus reflecting accurate rule-based processing but not human-like reasoning. Overall, LLMs are accurate semantic operators, but fail to capture the pragmatic enrichments characteristic of human reasoning. Crucially, LLM accuracy is neither predicted nor boosted by open vs. closed status, training orientation, or architecture type, suggesting that pragmatic reasoning is still an emerging ability in the cognitive toolkit of artificial systems.

Abstract

LLM Reasoning

Explore similar work

Tracing the ongoing emergence of human-like reasoning in Large Language Models

Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs

Humans and LLMs Diverge on Probabilistic Inferences