cs.AIOct 15, 2026
SaveAre Large Reasoning Models Good Everyday Reasoners? A Comparison with Non-Reasoning Models on Logical Reasoning Tasks
Organizations: Keio University
Abstract
Large reasoning models (LRMs) that leverage test-time computation achieve strong performance on mathematical and scientific benchmarks and are increasingly deployed in real-world applications. However, their ability to perform everyday logical reasoning, which often departs from textbook formal logic, remains underexplored. In this work, we systematically evaluate how well reasoning models align with everyday reasoning patterns by introducing a benchmark based on linguistic and pragmatic reasoning phenomena. We also compare reasoning models with non-reasoning models under controlled prompting conditions. We find that, in many cases, (1) reasoning models show degraded performance in everyday reasoning when such reasoning deviates from formal logical rules, (2) some non-reasoning models with chain-of-thought prompting outperform comparable reasoning models in everyday reasoning, and (3) prompting with additional instructions provides limited control over reasoning behavior. These findings suggest that current reasoning models do not reliably capture everyday reasoning and point to limitations in post-training, motivating improved training objectives and evaluation across diverse forms of reasoning.
Figures & tables
Figure 1: Taxonomy of reasoning patterns along everyday acceptability and formal validity. The highlighted off-diagonal cells correspond to gap cases .
| Label | Domain | Pattern | Everyday | Formal | Example |
| MP | Prop. | Yes | Yes | It is raining. If it rains, the ground is wet. Therefore, the ground is wet. | |
| MT | Prop. | Yes | Yes | The ground is not wet. If it rains, the ground is wet. Therefore, it is not raining. | |
| VT | Prop. | No | Yes | 2 + 2 does not equal 5. Therefore, if 2 + 2 equals 5, there is a blue car outside. | |
| Ross | Deon. | No | Yes | You are obliged to tell the truth. Therefore, you are obliged to tell the truth or lie. | |
| dFC | Deon. | Yes | No | You may have cake or fruit. Therefore, you may have cake. | |
| eFC | Epis. | Yes | No | It might be raining or snowing. Therefore, it might be raining. |
Table 1: Reasoning patterns across propositional, deontic, and epistemic domains. “Everyday” denotes everyday acceptability, while “Formal” denotes formal validity. denotes obligation, permission, epistemic possibility, and epistemic necessity.
Figure 2: Average alignment rates with respect to everyday acceptability, grouped by validity and acceptability of the reasoning patterns, for reasoning and non-reasoning models.
Figure 3: Average alignment rates by reasoning pattern. MP and MT are valid and acceptable patterns, VT and Ross are valid but unacceptable patterns, dFC , eFC , eSI , and dSI are invalid but acceptable patterns, and CI and eDE are invalid and unacceptable patterns.
Figure 4: Comparison of formalist bias (FB) scores across context conditions. Each subplot corresponds to a prompt type. Positive values indicate a preference for formal validity, while negative values indicate a preference for everyday acceptability. Colors distinguish reasoning and non-reasoning models.
| Reasoning model (qwen3-30b-a3b-thinking) | Non-reasoning model (qwen3-30b-a3b-instruct) + CoT |
| […] We can also think in terms of logical implication: A: P (where P = "Eiffel Tower is in London") B: P Q (where Q = "Socrates was born in Greece") We know that P is true. We want to know if P implies (P Q). We can use a truth table or logical equivalence: P Q is equivalent to P Q. So, B is equivalent to P Q. Now, if we have P (which is A), then P Q is true because P is true (so the disjunction is true regardless of Q). Therefore, A (which is P) implies B (which is P Q). Hence, B follows from A. Therefore, the answer is ’yes’. […] yes (not aligned with everyday reasoning) | […] However, A only asserts that recycling is the required action. It does not mention incineration at home, nor does it imply that incineration is acceptable or required. In fact, incinerating plastic at home is generally discouraged due to environmental and health risks, so it’s not a standard or morally acceptable alternative in most ethical frameworks. Moreover, the disjunction in B introduces a new action (incineration) that is not supported or implied by A. A does not say that recycling is optional or that other actions are permissible. So, from A, we cannot conclude that B is true. […] no (aligned with everyday reasoning) |
Table 2: Sample responses from the reasoning model (qwen3-30b-a3b-thinking-2507) on a VT pattern reasoning and the non-reasoning model + CoT (qwen3-30b-a3b-instruct-2507) on a Ross pattern reasoning. The reasoning model’s output focuses on the logical structure of the statements, leading to a conclusion not aligned with everyday reasoning, while the non-reasoning model’s output emphasizes the content and implications of the statements, leading to a conclusion aligned with everyday reasoning.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Zero-Shot (base) prompt example |
| ⬇ Determine whether statement B follows from statement A . Respond only with ’ yes ’, ’ no ’, or ’ undetermined ’, and nothing else . A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright . |
| CoT prompt example |
| ⬇ Determine whether statement B follows from statement A . Provide your reasoning step by step before giving the final answer . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright . |
Table 3: Prompt examples for Experiment 1.
| Formal Context prompt example (Non-Reasoning) |
| ⬇ Determine whether the statement B follows from statement A . Provide your reasoning step by step before giving the final answer . Answer using strict formal logic . Ignore conversational or pragmatic interpretations . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright . |
| Formal Context prompt example (Non-Reasoning + CoT) |
| ⬇ Determine whether the statement B follows from statement A . Provide your reasoning step by step before giving the final answer . Answer using strict formal logic . Ignore conversational or pragmatic interpretations . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright . |
| Everyday Context prompt example (Reasoning) |
| ⬇ Determine whether statement B follows from statement A . Answer based on everyday intuitive interpretation . Consider what people would naturally infer . Respond only with ’ yes ’, ’ no ’, or ’ undetermined ’, and nothing else . A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright . |
| Everyday Context prompt example (Non-Reasoning + CoT) |
Table 4: Additional prompt examples for Experiment 2.
Figure 5: Distribution of everyday-aligned, undetermined, and formal-aligned predictions by reasoning pattern. Blue, gray, and red bars correspond to everyday-aligned, undetermined, and formal-aligned predictions, respectively.
| Model | Alignment rate (std) |
| gpt-oss-20b | 100.0 (0.0) |
| gpt-oss-120b | 100.0 (0.0) |
| olmo-3.1-32b-think | 100.0 (0.0) |
| llama-3.3-70b-instruct + CoT | 99.2 (1.2) |
| qwen3-30b-a3b-thinking-2507 | 100.0 (0.0) |
| olmo-3.1-32b-instruct + CoT | 100.0 (0.0) |
Table 5: Alignment rates ( std) for each model on everyday acceptable and formally valid cases (%).
| Model | Alignment rate (std) |
| gpt-oss-20b | 3.3 (1.2) |
| gpt-oss-120b | 2.5 (0.0) |
| olmo-3.1-32b-think | 22.5 (2.0) |
| llama-3.3-70b-instruct + CoT | 25.0 (8.9) |
| qwen3-30b-a3b-thinking-2507 | 2.5 (2.0) |
| olmo-3.1-32b-instruct + CoT | 42.5 (2.0) |
Table 6: Alignment rates ( std) for each model on everyday unacceptable but formally valid cases (%).
| Model | Alignment rate (std) |
| gpt-oss-20b | 25.0 (2.7) |
| gpt-oss-120b | 12.1 (2.1) |
| olmo-3.1-32b-think | 34.6 (1.6) |
| llama-3.3-70b-instruct + CoT | 92.9 (2.1) |
| qwen3-30b-a3b-thinking-2507 | 40.0 (2.7) |
| olmo-3.1-32b-instruct + CoT | 47.9 (0.6) |
Table 7: Alignment rates ( std) for each model on everyday acceptable but formally invalid cases (%).
| Model | Alignment rate (std) |
| gpt-oss-20b | 100.0 (0.0) |
| gpt-oss-120b | 100.0 (0.0) |
| olmo-3.1-32b-think | 100.0 (0.0) |
| llama-3.3-70b-instruct + CoT | 80.8 (8.5) |
| qwen3-30b-a3b-thinking-2507 | 100.0 (0.0) |
| olmo-3.1-32b-instruct + CoT | 100.0 (0.0) |
Table 8: Alignment rates ( std) for each model on everyday unacceptable and formally invalid cases (%).
| Pattern | Reasoning | Non-reasoning (+ CoT) | Non-reasoning w/o CoT |
| MP | 100.0 (0.0) | 100.0 (0.0) | 100.0 (0.0) |
| MT | 100.0 (0.0) | 100.0 (0.0) | 100.0 (0.0) |
| Ross | 12.5 (18.9) | 50.0 (10.8) | 3.3 (2.4) |
| VT | 1.3 (2.2) | 40.0 (24.8) | 30.0 (42.4) |
| dFC | 36.3 (18.8) | 80.0 (18.7) | 95.0 (4.1) |
| eFC | 58.8 (20.1) | 96.7 (4.7) | 100.0 (0.0) |
Table 9: Average alignment rates ( std) with respect to everyday reasoning by reasoning pattern (%).
| Model | Base | CoT | Everyday | Formal | Meta |
| gpt-oss-20b | 0.617 (0.031) | n/a | -0.042 (0.051) | 0.992 (0.007) | 0.600 (0.014) |
| gpt-oss-120b | 0.750 (0.020) | n/a | -0.011 (0.016) | 0.972 (0.004) | 0.400 (0.062) |
| olmo-3.1-32b-think | 0.361 (0.020) | n/a | -0.500 (0.007) | 0.864 (0.024) | -0.483 (0.027) |
| qwen3-30b-a3b-think | 0.447 (0.020) | n/a | -0.519 (0.041) | 0.992 (0.012) | 0.222 (0.055) |
| llama-3.3-70b-in | -0.292 (0.012) | -0.406 (0.055) | -0.508 (0.056) | -0.128 (0.055) | -0.297 (0.010) |
| olmo-3.1-32b-in | -0.142 (0.000) | -0.122 (0.010) | -0.367 (0.035) | 0.844 (0.014) | -0.028 (0.008) |
Table 10: Formalist Bias (FB) scores across models and prompting conditions. Positive values indicate formal preference, negative values indicate everyday preference.
| Model | Prompt | FA | EA | FB | RR |
| gpt-oss-20b | Base | 0.794 (0.017) | 0.178 (0.016) | 0.617 (0.031) | 0.972 (0.010) |
| gpt-oss-20b | Everyday | 0.472 (0.028) | 0.514 (0.024) | -0.042 (0.051) | 0.986 (0.004) |
| gpt-oss-20b | Formal | 0.992 (0.007) | 0.000 (0.000) | 0.992 (0.007) | 0.992 (0.007) |
| gpt-oss-20b | Meta | 0.797 (0.010) | 0.197 (0.004) | 0.600 (0.014) | 0.994 (0.008) |
| gpt-oss-120b | Base | 0.839 (0.008) | 0.089 (0.014) | 0.750 (0.020) | 0.928 (0.010) |
| gpt-oss-120b | Everyday | 0.478 (0.010) | 0.489 (0.010) | -0.011 (0.016) | 0.967 (0.014) |
Table 11: Detailed gap-Based evaluation. FA and EA denote alignment with formal and everyday labels, respectively; FB (highlighted) is defined as ; RR denotes the proportion of cases where the model commits to either the formal or everyday label.
Figure 6: Annotated reasoning trace for Ross’s paradox where formal validity ( yes ) conflicts with everyday acceptability ( no ). The model prediction aligns with everyday reasoning, and the trace is annotated with both everyday and mixed labels.
Figure 7: Annotated reasoning trace for an epistemic free choice permission (eFC) inference where formal validity ( no ) diverges from everyday acceptability ( yes ). The model prediction aligns with everyday reasoning. The trace is annotated with both formal and mixed labels.
Figure 8: Example of a deontic scalar implicature (dSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts undetermined , and the reasoning trace is annotated as undetermined .
Figure 9: Example of a deontic scalar implicature (dSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts no , and the reasoning trace is annotated as mixed .
Figure 10: Example of a vacuous truth (VT) case where formal validity ( yes ) diverges from everyday acceptability ( no ). The model predicts yes , and the reasoning trace is annotated as formal .
Figure 11: Example of a deontic free choice (dFC) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts yes , and the reasoning trace is annotated as everyday and mixed .
Figure 12: Example of an epistemic scalar implicature (eSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts yes , and the reasoning trace is annotated as mixed .
Figure 13: Example of an epistemic scalar implicature (eSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts no , and the reasoning trace is annotated as formal .