DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models

Amit Dhanda

cs.AIApr 3, 2026

DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models

Authors: Amit Dhanda

Organizations: Amazon

Abstract

Reasoning benchmarks typically evaluate whether a model derives the correct answer from a fixed premise set, but they under-measure a closely related capability that matters in dynamic environments: belief revision under minimal evidence change. We introduce DeltaLogic, a benchmark transformation protocol that converts natural-language reasoning examples into short revision episodes. Each episode first asks for an initial conclusion under premises P, then applies a minimal edit δ(P), and finally asks whether the previous conclusion should remain stable or be revised. We instantiate DeltaLogic from FOLIO and ProofWriter and evaluate small causal language models with constrained label scoring. On a completed 30-episode Qwen evaluation subset, stronger initial reasoning still does not imply stronger revision behavior: Qwen3-1.7B reaches 0.667 initial accuracy but only 0.467 revision accuracy, with inertia rising to 0.600 on episodes where the gold label should change, while Qwen3-0.6B collapses into near universal abstention. There, Qwen3-4B preserves the same inertial failure pattern (0.650 initial, 0.450 revised, 0.600 inertia), whereas Phi-4-mini-instruct is substantially stronger (0.950 initial, 0.850 revised) but still exhibits non-trivial abstention and control instability. These results suggest that logical competence under fixed premises does not imply disciplined belief revision after local evidence edits. DeltaLogic therefore targets a distinct and practically important reasoning capability that complements existing logical inference and belief-updating benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: DeltaLogic construction pipeline. A standard reasoning item is turned into a minimally edited revision episode with a known semantic effect, which makes revision errors measurable rather than anecdotal.

Model	Slice $n$	Init. $\uparrow$	Rev. $\uparrow$	Inertia $\downarrow$	Over-flip $\downarrow$	Abstain $\downarrow$
Qwen3-0.6B	30	0.300	0.400	0.400	0.000	1.000
Qwen3-1.7B	30	0.667	0.467	0.600	0.000	0.000
Qwen3-4B	20	0.650	0.450	0.600	0.000	0.000
Phi-4-mini	20	0.950	0.850	0.200	0.100	0.350

Table 1: Main DeltaLogic results on the completed generative-model evaluation subsets.

Panel of Figure 2 — Figure 2: Failure-mode comparison across completed runs. The Qwen family remains inertia-dominated, while Phi-4-mini trades lower inertia for non-zero over-flip and abstention.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Aug 31, 2026

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.

Qiyao Yan, Chenpeng Wang, Liangming Pan

State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1

Language Modeling Language Model Calibration Implicit Reasoning in Language Models LLM Evaluation LLM Reasoning Language Model Probing+1

Ask PDF

May 23, 2026

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason about parameter-dependent dynamics. We present ChaosBench-Logic v2, a 40,886-question benchmark over 165 dynamical systems with 27 FOL predicates and 78 axiom edges, together with CARE (Calibration- and Adversarial-Robust Evaluation), a protocol that surfaces these pathologies. Evaluating 14 models, we find that regime-transition reasoning remains near random (MCC = 0.05) even for frontier models, whereas FOL deduction with given premises reaches MCC = 0.52. Per-family decomposition shows that the proprietary-model advantage concentrates on cross-indicator (+0.40) and consistency tasks, while open-source Qwen 2.5-32B dominates indicator diagnostics (0.91 vs. 0.45). Two models exhibit negative MCC on bifurcation questions, confirmed as systematic anti-correlation via confusion-matrix analysis.

Noel Thomas

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

LLM Evaluation Language Model Robustness LLM Reasoning Large Reasoning Models

Ask PDF

Jul 16, 2026

Controlled Reformulation Testing for Logical Consistency in Large Language Models

Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs' ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves

98.9\%

base accuracy but only

60.3\%

family-level consistency, while reasoning-optimized o4-mini achieves

96.9\%

consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting (

72.4\%

for GPT-5.4-mini) and double negation (

84.6\%

), while surface-level rephrasing remains robust (

94-100\%

). Increasing reasoning effort improves GPT-5.4-mini to

85.4\%

consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.

Alexander Gu, Alan Chen

University of Texas at Austin

LLM Evaluation Reasoning Consistency in Language Models Language Model Robustness Self-Consistency in Language Models LLM Reasoning

Ask PDF

Abstract

LLM Reasoning

Explore similar work

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

Controlled Reformulation Testing for Logical Consistency in Large Language Models