Large reasoning models (LRMs) that leverage test-time computation achieve strong performance on mathematical and scientific benchmarks and are increasingly deployed in real-world applications. However, their ability to perform everyday logical reasoning, which often departs from textbook formal logic, remains underexplored. In this work, we systematically evaluate how well reasoning models align with everyday reasoning patterns by introducing a benchmark based on linguistic and pragmatic reasoning phenomena. We also compare reasoning models with non-reasoning models under controlled prompting conditions. We find that, in many cases, (1) reasoning models show degraded performance in everyday reasoning when such reasoning deviates from formal logical rules, (2) some non-reasoning models with chain-of-thought prompting outperform comparable reasoning models in everyday reasoning, and (3) prompting with additional instructions provides limited control over reasoning behavior. These findings suggest that current reasoning models do not reliably capture everyday reasoning and point to limitations in post-training, motivating improved training objectives and evaluation across diverse forms of reasoning.
Figures & tables
Figure 1: Taxonomy of reasoning patterns along everyday acceptability and formal validity. The highlighted off-diagonal cells correspond to gap cases .
Label
Domain
Pattern
Everyday
Formal
Example
MP
Prop.
A,A→B⇒B
Yes
Yes
It is raining. If it rains, the ground is wet. Therefore, the ground is wet.
MT
Prop.
¬B,A→B⇒¬A
Yes
Yes
The ground is not wet. If it rains, the ground is wet. Therefore, it is not raining.
VT
Prop.
¬A⇒A→B
No
Yes
2 + 2 does not equal 5. Therefore, if 2 + 2 equals 5, there is a blue car outside.
Ross
Deon.
OA⇒O(A∨B)
No
Yes
You are obliged to tell the truth. Therefore, you are obliged to tell the truth or lie.
dFC
Deon.
P(A∨B)⇒PA
Yes
No
You may have cake or fruit. Therefore, you may have cake.
eFC
Epis.
◊(A∨B)⇒◊A
Yes
No
It might be raining or snowing. Therefore, it might be raining.
Table 1: Reasoning patterns across propositional, deontic, and epistemic domains. “Everyday” denotes everyday acceptability, while “Formal” denotes formal validity. O denotes obligation, P permission, ◊ epistemic possibility, and □ epistemic necessity.
Figure 2: Average alignment rates with respect to everyday acceptability, grouped by validity and acceptability of the reasoning patterns, for reasoning and non-reasoning models.
Figure 3: Average alignment rates by reasoning pattern. MP and MT are valid and acceptable patterns, VT and Ross are valid but unacceptable patterns, dFC , eFC , eSI , and dSI are invalid but acceptable patterns, and CI and eDE are invalid and unacceptable patterns.
Figure 4: Comparison of formalist bias (FB) scores across context conditions. Each subplot corresponds to a prompt type. Positive values indicate a preference for formal validity, while negative values indicate a preference for everyday acceptability. Colors distinguish reasoning and non-reasoning models.
Reasoning model (qwen3-30b-a3b-thinking)
Non-reasoning model (qwen3-30b-a3b-instruct) + CoT
[…] We can also think in terms of logical implication: A: ¬ P (where P = "Eiffel Tower is in London") B: P → Q (where Q = "Socrates was born in Greece") We know that ¬ P is true. We want to know if ¬ P implies (P → Q). We can use a truth table or logical equivalence: P → Q is equivalent to ¬ P ∨ Q. So, B is equivalent to ¬ P ∨ Q. Now, if we have ¬ P (which is A), then ¬ P ∨ Q is true because ¬ P is true (so the disjunction is true regardless of Q). Therefore, A (which is ¬ P) implies B (which is ¬ P ∨ Q). Hence, B follows from A. Therefore, the answer is ’yes’. […] yes (not aligned with everyday reasoning)
[…] However, A only asserts that recycling is the required action. It does not mention incineration at home, nor does it imply that incineration is acceptable or required. In fact, incinerating plastic at home is generally discouraged due to environmental and health risks, so it’s not a standard or morally acceptable alternative in most ethical frameworks. Moreover, the disjunction in B introduces a new action (incineration) that is not supported or implied by A. A does not say that recycling is optional or that other actions are permissible. So, from A, we cannot conclude that B is true. […] no (aligned with everyday reasoning)
Table 2: Sample responses from the reasoning model (qwen3-30b-a3b-thinking-2507) on a VT pattern reasoning and the non-reasoning model + CoT (qwen3-30b-a3b-instruct-2507) on a Ross pattern reasoning. The reasoning model’s output focuses on the logical structure of the statements, leading to a conclusion not aligned with everyday reasoning, while the non-reasoning model’s output emphasizes the content and implications of the statements, leading to a conclusion aligned with everyday reasoning.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Zero-Shot (base) prompt example
⬇ Determine whether statement B follows from statement A . Respond only with ’ yes ’, ’ no ’, or ’ undetermined ’, and nothing else . A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright .
CoT prompt example
⬇ Determine whether statement B follows from statement A . Provide your reasoning step by step before giving the final answer . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright .
Appendix
Table 3: Prompt examples for Experiment 1.
Formal Context prompt example (Non-Reasoning)
⬇ Determine whether the statement B follows from statement A . Provide your reasoning step by step before giving the final answer . Answer using strict formal logic . Ignore conversational or pragmatic interpretations . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright .
Formal Context prompt example (Non-Reasoning + CoT)
⬇ Determine whether the statement B follows from statement A . Provide your reasoning step by step before giving the final answer . Answer using strict formal logic . Ignore conversational or pragmatic interpretations . At the end of your response , include a single final line that contains only one word : ’ yes ’, ’ no ’, or ’ undetermined ’. A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright .
Everyday Context prompt example (Reasoning)
⬇ Determine whether statement B follows from statement A . Answer based on everyday intuitive interpretation . Consider what people would naturally infer . Respond only with ’ yes ’, ’ no ’, or ’ undetermined ’, and nothing else . A : If the sun rises , then the sky becomes bright . The sun rises . B : The sky becomes bright .
Everyday Context prompt example (Non-Reasoning + CoT)
Appendix
Table 4: Additional prompt examples for Experiment 2.
Figure 5: Distribution of everyday-aligned, undetermined, and formal-aligned predictions by reasoning pattern. Blue, gray, and red bars correspond to everyday-aligned, undetermined, and formal-aligned predictions, respectively.
Model
Alignment rate (std)
gpt-oss-20b
100.0 (0.0)
gpt-oss-120b
100.0 (0.0)
olmo-3.1-32b-think
100.0 (0.0)
llama-3.3-70b-instruct + CoT
99.2 (1.2)
qwen3-30b-a3b-thinking-2507
100.0 (0.0)
olmo-3.1-32b-instruct + CoT
100.0 (0.0)
Appendix
Table 5: Alignment rates ( ± std) for each model on everyday acceptable and formally valid cases (%).
Model
Alignment rate (std)
gpt-oss-20b
3.3 (1.2)
gpt-oss-120b
2.5 (0.0)
olmo-3.1-32b-think
22.5 (2.0)
llama-3.3-70b-instruct + CoT
25.0 (8.9)
qwen3-30b-a3b-thinking-2507
2.5 (2.0)
olmo-3.1-32b-instruct + CoT
42.5 (2.0)
Appendix
Table 6: Alignment rates ( ± std) for each model on everyday unacceptable but formally valid cases (%).
Model
Alignment rate (std)
gpt-oss-20b
25.0 (2.7)
gpt-oss-120b
12.1 (2.1)
olmo-3.1-32b-think
34.6 (1.6)
llama-3.3-70b-instruct + CoT
92.9 (2.1)
qwen3-30b-a3b-thinking-2507
40.0 (2.7)
olmo-3.1-32b-instruct + CoT
47.9 (0.6)
Appendix
Table 7: Alignment rates ( ± std) for each model on everyday acceptable but formally invalid cases (%).
Model
Alignment rate (std)
gpt-oss-20b
100.0 (0.0)
gpt-oss-120b
100.0 (0.0)
olmo-3.1-32b-think
100.0 (0.0)
llama-3.3-70b-instruct + CoT
80.8 (8.5)
qwen3-30b-a3b-thinking-2507
100.0 (0.0)
olmo-3.1-32b-instruct + CoT
100.0 (0.0)
Appendix
Table 8: Alignment rates ( ± std) for each model on everyday unacceptable and formally invalid cases (%).
Pattern
Reasoning
Non-reasoning (+ CoT)
Non-reasoning w/o CoT
MP
100.0 (0.0)
100.0 (0.0)
100.0 (0.0)
MT
100.0 (0.0)
100.0 (0.0)
100.0 (0.0)
Ross
12.5 (18.9)
50.0 (10.8)
3.3 (2.4)
VT
1.3 (2.2)
40.0 (24.8)
30.0 (42.4)
dFC
36.3 (18.8)
80.0 (18.7)
95.0 (4.1)
eFC
58.8 (20.1)
96.7 (4.7)
100.0 (0.0)
Appendix
Table 9: Average alignment rates ( ± std) with respect to everyday reasoning by reasoning pattern (%).
Table 11: Detailed gap-Based evaluation. FA and EA denote alignment with formal and everyday labels, respectively; FB (highlighted) is defined as FA−EA ; RR denotes the proportion of cases where the model commits to either the formal or everyday label.
Figure 6: Annotated reasoning trace for Ross’s paradox where formal validity ( yes ) conflicts with everyday acceptability ( no ). The model prediction aligns with everyday reasoning, and the trace is annotated with both everyday and mixed labels.
Figure 7: Annotated reasoning trace for an epistemic free choice permission (eFC) inference where formal validity ( no ) diverges from everyday acceptability ( yes ). The model prediction aligns with everyday reasoning. The trace is annotated with both formal and mixed labels.
Figure 8: Example of a deontic scalar implicature (dSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts undetermined , and the reasoning trace is annotated as undetermined .
Figure 9: Example of a deontic scalar implicature (dSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts no , and the reasoning trace is annotated as mixed .
Figure 10: Example of a vacuous truth (VT) case where formal validity ( yes ) diverges from everyday acceptability ( no ). The model predicts yes , and the reasoning trace is annotated as formal .
Figure 11: Example of a deontic free choice (dFC) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts yes , and the reasoning trace is annotated as everyday and mixed .
Figure 12: Example of an epistemic scalar implicature (eSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts yes , and the reasoning trace is annotated as mixed .
Figure 13: Example of an epistemic scalar implicature (eSI) case where formal validity ( no ) diverges from everyday acceptability ( yes ). The model predicts no , and the reasoning trace is annotated as formal .
Large Reasoning Models (LRMs) improve performance by generating explicit intermediate reasoning traces through increased test-time compute, yet the assumption that longer reasoning is consistently beneficial remains under-examined. While recent evidence shows that additional reasoning can lead models to overthink, we ask: "Once a model has reached the correct answer, does further reasoning refine the solution, or deviate from it?" To study the dynamics after correctness, we introduce a prefix-level trajectory evaluation protocol grounded in reasoning sufficiency, defining the minimum reasoning budget required for a model to first generate the correct answer. This allows us to disentangle verbose overthinking, where additional reasoning is redundant but harmless, from harmful overthinking, where continued reasoning destabilizes an already-correct trajectory. Starting from multimodal benchmarks, we find that many instances considered reasoning-intensive require surprisingly little reasoning. Moreover, stopping at the first correct prefix improves accuracy over standard reasoning up to 21%, revealing that current models are limited not only by their ability to reason, but also by their inability to stop at the right time. Furthermore, while common efficiency strategies like early stopping substantially reduce verbose overthinking (up to 50%), they fail to mitigate harmful overthinking. Failure analysis reveals that correctness deviations are mainly driven by logical drift and visual reinterpretation. Finally, we show that our findings generalize to language-only reasoning benchmarks, highlighting harmful overthinking as a broader reliability risk. Code available at https://simonecaldarella.github.io/thinking-past-the-answer.
Simone Caldarella, Davide Talon, Rahaf Aljundi +2
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.
Contemporary large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these reasoning skills to more general and broader contexts--often termed general reasoning--remains under-explored. Unlike domain-specific reasoning, general reasoning relies less on expert knowledge but still presents formidable reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks. These results suggest that the reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code, Dataset, and Leaderboard: https://general365.github.io
Junlin Liu, Shengnan An, Shuang Zhou +10
University of Chinese Academy of Sciences · Meituan