The Logic Chain Breaking Point: A Controlled-Variable Benchmark for Reasoning Degradation in Open-Weight Language Models
Organizations: Department of Computer Science and Engineering, University of Rajshahi, Rajshahi, 6205, Bangladesh · Department of Computer Science and Engineering, Varendra University, Rajshahi, Bangladesh
Abstract
We introduce The Logic Chain Breaking Point, a controlled-variable benchmark for deductive reasoning in open-weight language models. It spans three phases: (i) pure modus-ponens chain scaling, (ii) distractor resilience with a fixed core chain, and (iii) contradiction detection under primacy versus recency placement. Independent variables follow a powers-of-two sequence , with ground truth known by construction. We evaluate sixteen open-weight models across instruction-tuned and reasoning-tuned panels, including a small-model suite and a larger local-model extension. All models are evaluated with greedy decoding under model-appropriate context and output budgets, while context- and length-limited trials are quarantined from the valid accuracy denominator. Our results reveal a sharply non-monotonic difficulty surface: pure deductive chains up to remain near saturation for most models, and distractor noise causes only model-specific degradation at high , while the inconsistent premise condition has the lowest accuracy. Instruction-tuned models, including larger 24B--27B baselines, score essentially zero on inconsistent premise sets, while reasoning-tuned models recover only partially and remain below reliable accuracy. Within this benchmark, inconsistent premise sets are associated with the strongest degradation under the benchmark's abstention-based scoring rule.
Figures & tables
| Model | Context | Max out | Type | Size |
| mistral-small3.2:24b | 8192 | 128 | Instruct | Large |
| gemma3:27b-it-qat | 8192 | 128 | Instruct | Large |
| gpt-oss:20b | 32768 | 8192 | Reasoning | Large |
| deepseek-r1:14b | 32768 | 8192 | Reasoning | Large |
| phi4-reasoning:plus | 32768 | 8192 | Reasoning | Large |
| google.gemma-4-12b-it | 8192 | 128 | Instruct | Small |
| Model | P1 ( ) | P2 ( ) | P3 Acc. [95% CI] | Size |
| Reasoning models | ||||
| gpt-oss:20b | NR | NR | 0.725 [0.57, 0.84] | Large |
| deepseek-r1:14b | 128 | 64 | 0.525 [0.38, 0.67] | Large |
| phi4-reasoning:plus | NR | NR | 0.475 [0.33, 0.63] | Large |
| deepseek-r1-distill-llama-8b | 128 | 128 | 0.725 [0.57, 0.84] | Small |
| deepseek-r1-distill-qwen-7b | NR | 128 | 0.625 [0.47, 0.76] | Small |