Compositional Consistency-Guided Decoding for Three-Way Logical Question Answering

Tianyi Huang; Ming Hou; Jiaheng Su; Yutong Zhang; Ziling Zhang

cs.CLMar 12, 2026

Compositional Consistency-Guided Decoding for Three-Way Logical Question Answering

Authors: Tianyi Huang, Ming Hou, Jiaheng Su, Yutong Zhang, Ziling Zhang

Organizations: Ryquo · App-In Club

Abstract

Three-way logical question answering (QA) assigns one of $\text{True}$ , $\text{False}$ , or $\text{Unknown}$ to a hypothesis $H$ given a premise set $S$ . We study this task as a compact compositional inference problem: predictions for $H$ and for a mechanically negated hypothesis $\neg H$ should agree under a deterministic negation map. Despite this simple structure, large language models (LLMs) can exhibit two practical failure modes: (i) negation inconsistency, where answers to $H$ and $\neg H$ violate the required label mapping, and (ii) epistemic $\text{Unknown}$ , where the model abstains even when one side is entailed. We introduce CGD-PD, a lightweight, training-free test-time layer that combines neural 3-way classification, symbolic negation-consistency projection, and targeted binary entailment probes. On one validation split of FOLIO's first-order logic fields, CGD-PD improves accuracy by 4.4 points on GPT-5.2 and 6.8 points on Claude Sonnet 4.5, while reducing $\text{Unknown}$ predictions and epistemic abstention. These results provide a controlled proof of concept that simple logical composition at inference time can help evaluate and improve LLM reasoning reliability; they do not, by themselves, establish robustness beyond this formal benchmark setting.

Figures & tables

Model	Method	Acc. (%)	Unknown (%)	Epist. Unknown (%)	Calls
GPT-5.2	Single	63.7	57.4	41.5	1.00
GPT-5.2	CGD-PD	68.1	53.9	36.3	4.36
Claude Sonnet 4.5	Single	42.2	75.5	72.6	1.00
Claude Sonnet 4.5	CGD-PD	49.0	58.8	53.3	4.91

Table 1: FOLIO validation results on FOL fields (204 validation examples). Accuracy, Unknown rate, and epistemic Unknown rate are percentages; epistemic Unknown is the fraction of gold True / False examples predicted as Unknown . “Calls” is the mean number of wrapper model invocations per example.

Model	Method	Cov.	Ans. acc.	Gold U kept	Gold U $\to$ T	Gold U $\to$ F
GPT-5.2	Single	42.6	79.3	88.4	3	5
GPT-5.2	CGD-PD	46.1	83.0	88.4	4	4
Claude Sonnet 4.5	Single	24.5	60.0	81.2	9	4
Claude Sonnet 4.5	CGD-PD	41.2	61.9	69.6	13	8

Table 2: Coverage and decisive-label reliability on FOLIO validation (204 examples). Coverage, answered accuracy, and Gold U kept are percentages; Gold U

\to

T/F are counts of genuinely underspecified examples predicted as decisive labels. T/F/U denote True , False , and Unknown .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Row-normalized confusion matrices on the FOLIO validation split. For each model, the left heatmap is the single-call classifier and the right heatmap is CGD-PD; rows are gold labels and columns are predicted labels. Each cell shows the row-normalized rate with the raw count in parentheses.

Premise excerpt	Hypothesis $H$	Gold	Single	Ours	Signals used by our method
The Picuris Mountains are a mountain range in New Mexico or Texas.	The Picuris Mountains are in Texas.	U	F	U	Probe labels: $y(H)=\text{{Unknown}}$ and $y(\neg H)=\text{{Unknown}}$ . CGD-PD then runs proof-driven entailment checks and finds $\mathsf{EntailsYesNo}(S,H)=\text{{No}}$ and $\mathsf{EntailsYesNo}(S,\neg H)=\text{{No}}$ , so it preserves Unknown . This illustrates that the wrapper need not force a decisive answer when neither side is supported.
The Picuris Mountains are a mountain range in New Mexico or Texas.	The Picuris Mountains are in New Mexico.	F	T	F	Probe labels: $y(H)=\text{{Unknown}}$ and $y(\neg H)=\text{{True}}$ . Because the negated form is decisive, CGD-PD maps that decision back through the negation constraint and returns False . This illustrates how asymmetric evidence from logically linked probes can correct a single-prompt error.

Appendix

Table 3: Representative GPT-5.2 validation cases illustrating two mechanisms of CGD-PD. The first case preserves a genuine Unknown when neither side is entailed; the second uses the decisive negated probe to correct the final label for

H

. T / F / U denote True / False / Unknown .

		Train (500)			Validation (204)
Model	Method	Acc	Unk	Calls/ex	Acc	Unk	Calls/ex
GPT-5.2	Single	0.632	0.544	1.00	0.637	0.574	1.00
GPT-5.2	gate3	0.632	0.568	3.00	–	–	–
GPT-5.2	CGR3 (3 calls)	0.654	0.454	3.00	–	–	–
GPT-5.2	CGD-PD (final)	0.682	0.396	3.85	0.681	0.539	4.36
Claude Sonnet 4.5	Single	0.436	0.744	1.00	0.422	0.755	1.00

Appendix

Table 4: Ablations and negative results. Acc = accuracy; Unk = Unknown rate; Calls/ex = average model calls per example. Validation contains 204 examples in FOLIO v0.0, so --max_examples is an upper bound rather than an exact size control.

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 260 papers

Jun 19, 2026

Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning

Large language models (LLMs) deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vacuously achieved through systematic abstention. A model that withholds commitment to either entailment or refutation satisfies negation consistency while providing no utility. We introduce Coherence Under Commitment (CUC), a dual-query evaluation paradigm that jointly measures consistency and decisiveness. CUC contributes three innovations: (1) a commitment score

c(\varphi) = p(\varphi) + p(\lnot\varphi)

quantifying probability mass allocated to decisive outcomes; (2) a deterministic elicitation protocol via normalized YES/NO log probabilities, eliminating sampling variance; and (3) a 3-way decision framework (True/False/Uncertain) operationalizing the coherence-commitment trade-off into metrics. Experiments on four open-weight LLMs (1B-3B) across 204 FOLIO examples expose a sharp frontier. Qwen2.5-3B achieves near-zero contradiction (

\mathbb{E}[v_{\mathrm{neg}}]{=}0.025

) but only

7.4\%

coverage, while TinyLlama-1.1B reaches

79.4\%

coverage with violations on every example. Coherence-only evaluation would rank the abstaining model first; CUC exposes this as vacuous, and the frontier generalizes to LogiQA~v2 (

ρ{=}0.97

). We argue that evaluation must report both coherence and non-vacuous commitment and release a toolkit for standardized assessment.

Noor Islam S. Mohammad, Mahmudul Hasan

Department of Computer Science, Informatics Institute, Istanbul Technical University, ˙Istanbul, Türkiye · School of Information Technology, Deakin University, Geelong, VIC 3220, Australia

LLM Evaluation Reasoning Consistency in Language Models LLM Abstention LLM Reasoning

Ask PDF

Jul 16, 2026

Controlled Reformulation Testing for Logical Consistency in Large Language Models

Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs' ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves

98.9\%

base accuracy but only

60.3\%

family-level consistency, while reasoning-optimized o4-mini achieves

96.9\%

consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting (

72.4\%

for GPT-5.4-mini) and double negation (

84.6\%

), while surface-level rephrasing remains robust (

94-100\%

). Increasing reasoning effort improves GPT-5.4-mini to

85.4\%

consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.

Alexander Gu, Alan Chen

University of Texas at Austin

LLM Evaluation Reasoning Consistency in Language Models Language Model Robustness Self-Consistency in Language Models LLM Reasoning

Ask PDF

Aug 13, 2026

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.6 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.

Obed Junias, Maria Leonor Pacheco

University of Colorado Boulder

Compositional Reasoning Multiple-Choice Question Answering Question Answering Structured Reasoning LLM Reasoning Integer Linear Programming

Ask PDF

Abstract

Question Answering

Explore similar work

Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning

Controlled Reformulation Testing for Logical Consistency in Large Language Models

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options