Explanation Generation for Contradiction Reconciliation with LLMs

Jason Chan; Zhixue Zhao; Robert Gaizauskas

cs.CLMar 24, 2026

Explanation Generation for Contradiction Reconciliation with LLMs

Authors: Jason Chan, Zhixue Zhao, Robert Gaizauskas

Organizations: University of Sheffield, UK

Abstract

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social and professional settings is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" appear contradictory, but are in fact compatible if Cassie was assigned the daily chore of buying coffee for all her coworkers. Despite advances in the reasoning capabilities of large language models (LLMs), their ability to hypothesize such explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of reusing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 13 LLMs in thinking and non-thinking modes reveal models' limited success in this task, with longer reasoning chains negatively associated with explanation success. As a proof of concept, we further adapt our introduced task and metrics for reinforcement-learning training, finding that trained models exhibit improved accuracy in non-thinking mode on two narrative reasoning benchmarks (True Detective and MuSR). Our findings highlight reconciliatory explanation generation as a challenge in LLM reasoning and motivate further work to investigate its potential for improving model capabilities.

Figures & tables

Figure 1: Given a premise ( P ) and hypothesis ( H ) that a judge model deems contradictory, our novel task requires models to generate a successful explanation ( E ) such that H is judged as entailed by P combined with E .

Figure 2: “ Contradiction ” label weight (

w_{con.}

) of each instance (i.e. proportion of annotators that labeled it as “ Contradiction ”) in the ChaosNLI-MNLI-C subset (275 instances in total).

	Candidate Judge Model
Qwen3.5	Qwen3.5-0.8B	59.17
	Qwen3.5-2B	62.28
	Qwen3.5-4B	88.58
	Qwen3.5-9B	91.35
	Qwen3.5-27B	93.08
Gemma-4	Gemma-4-E2B-it	74.74

Table 1: Accuracy of candidate judge models (all tested in non-thinking mode) in predicting the most frequently assigned label of instances in ChaosNLI-MNLI with at least 80% annotator agreement. Score of our chosen judge, Gemma-4-31B-it, is enclosed in blue .

	Source Model of Generated Explanations	Coherence		Effectiveness		Explanation Success
	Source Model of Generated Explanations	Non- Thinking	Thinking	Non- Thinking	Thinking	Non- Thinking	Thinking
Qwen 3.5	Qwen3.5-0.8B	72.31	72.82	26.67	22.05	11.28	16.41
	Qwen3.5-2B	66.15	66.15	38.46	57.95	18.46	33.33
	Qwen3.5-4B	76.41	74.36	17.95	57.44	13.33	40.51
	Qwen3.5-9B	86.67	81.54	30.26	64.10	24.62	51.28
	Qwen3.5-27B	75.90	80.51	61.54	70.77	44.10	54.36

Table 2: Coherence, effectiveness, and success rate of each model’s generated explanations (in %), as evaluated by Gemma-4-31B-it as the judge model (enclosed in blue ). Scores in the row corresponding to Gemma-4-31B-it are when the model itself acts as both explanation generator and the judge doing the evaluating. Highest score in each column is in bold . Appendix J includes an extended version of this table with 95% bootstrap confidence intervals.

	Source Model of Generated Explanations	mean # of output tokens
		successful explanations	unsuccessful explanations
Qwen3.5	Qwen3.5-0.8B	6211.84	6470.07
	Qwen3.5-2B	5757.32	6173.87
	Qwen3.5-4B	5596.57	6153.34
	Qwen3.5-9B	5324.74	5409.26
	Qwen3.5-27B	4905.58	5542.65

Table 3: Comparison of mean output tokens (including reasoning tokens) generated by each model in thinking mode, for successful and unsuccessful explanations.

		Effectiveness $M_{NLI}(p;e,h)$
		con.	neu.	ent.
Coherence $M_{NLI}(p,e)$	con.	-1	-1	-1
	neu.	-1	0	0.5
	ent.	-1	0	1

Table 4: Reward matrix for RL, based on NLI judge’s evaluation of explanation coherence

M_{NLI}(p,e)

and effectiveness

M_{NLI}(p;e,h)

. Unlike task metrics used in evaluation, we introduce negative (-1) and partial (0.5) rewards based on specific NLI label combinations.

Model	Mode	True Detective	MuSR
Baseline	Non-thinking	22.51	38.40
Baseline	Thinking	25.65	57.20
DAPO-trained
$\epsilon_{\mathrm{high}}=0.20$	Non-thinking	23.21	43.47
$\epsilon_{\mathrm{high}}=0.28$		24.61	40.53
$\epsilon_{\mathrm{high}}=0.36$		23.73	43.20

Table 5: Accuracy (%) on True Detective and MuSR of the baseline Qwen3.5-2B (in non-thinking and thinking modes) and DAPO-trained models (only in non-thinking mode) with different upper clipping thresholds

e_{high}

. Accuracy of DAPO-trained models are mean values across three training runs with same hyperparameters and different random seeds. Highest score in each column is shown in bold and second highest underlined .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3.5 Qwen Team (2026)
Full model name	Parameter count
Qwen3.5-0.8B	$8.0\times 10^{8}$
Qwen3.5-2B	$2.0\times 10^{9}$
Qwen3.5-4B	$4.0\times 10^{9}$
Qwen3.5-9B	$9.0\times 10^{9}$
Qwen3.5-27B	$2.7\times 10^{10}$

Appendix

Table 6: Details of the 13 models used in our experiments. All models can be prompted in both thinking and non-thinking modes.

Candidate Judge Model	Agreement Threshold ( $t$ )
	0.0 *	0.7	0.8	0.9
Qwen3.5-0.8B	43.03	52.07	59.17	70.18
Qwen3.5-2B	48.03	57.07	62.28	66.67
Qwen3.5-4B	69.29	85.17	88.58	96.49
Qwen3.5-9B	69.61	85.00	91.35	100.00
Qwen3.5-27B	72.61	90.17	93.08	100.00

Appendix

Table 7: Accuracy (%) of candidate NLI judge models on ChaosNLI-MNLI instances, filtered by different annotator agreement thresholds (

t

). All models are prompted in non-thinking mode. The row corresponding to our selected judge, Gemma-4-31B-It is in bold . * Where

t=0.0

, the full unfiltered dataset is used.

Premise: In the depths of the Cold War, many Americans suspected Communists had infiltrated Washington and were about to subvert our democracy. Hypothesis: Communists assisted America’s government during the Cold War. (57 out of 100 annotators labeled as contradiction)
Source	Adversarial Explanation
gpt-5.6-terra	Cold War debates in Washington often blurred the line between government service and fears of political subversion.
Manually written	As such, they were eager to assist America’s government by reporting any suspected Communists.

Appendix

Table 8: An example premise-hypothesis pair for which adversarial explanations, seemingly relevant but ineffective, are created through (a) prompting gpt-5.6-terra; and (b) manual writing.

Source of adversarial explanations	C	E	S
Model-generated (gpt-5.6-terra)	99.49	2.05	2.05
Manually written	100.00	6.00	6.00

Appendix

Table 9: Coherence ( C ), effectiveness ( E ) and success rate ( S ) as predicted by our selected judge, Gemma-4-31B-it, when the model is tasked with evaluating adversarial explanations that are deliberately crafted to appear relevant but ineffective in reconciling the premise and hypothesis.

Model	Explanation	E / C / S
Ex 1 ( $w_{con}$ : 0.64) \| P: Because marginal costs are very low, a newspaper price for preprints might be as low as 5 or 6 cents per piece. H: Newspaper preprints can cost as much as $5.
Q2B-n Qwen3.5-2B non-thinking	Newspaper preprints can cost as much as $5 because marginal costs are very low, allowing for a price point that is both affordable and sustainable. (32 tokens)	✓ / $\times$ / $\times$
Q2B-t Qwen3.5-2B thinking	Reasoning trace: 1. Analyze the Request:[…] 2. Analyze the Contradiction: […] 3. Find the Reconciliation […] Likely explanation: There is a difference between the unit price and total cost. Perhaps there is a subscription, shipping fee, bulk discount? […] Alternative interpretation: Maybe the $5 figure refers to a premium edition or shipping included. […] Wait, I must check if there are any hidden tricks. […] Although the per-unit cost is minimal, buying a large number of copies means the overall expense reaches$ 5. (4984 tokens)	✓ / ✓ / ✓
5.6Sol-n GPT-5.6-sol non-thinking	The low price applies to individual copies , while a bundle of roughly 100 preprints could cost about $5. (27 tokens)	✓ / ✓ / ✓
Ex 2 ( $w_{con}$ : 0.79) \| P: What Ellison is doing here, as Hemingway did, is equating the process of becoming an artist with that of becoming a man. H: Ellison and Hemingway took different ways to compare becoming a man.
G2B-n Gemma-4-E2B-it non-thinking	Ellison’s focus on artistic creation reflects a specific facet of the broader human journey that Hemingway addressed through his own literary explorations. (27 tokens)	$\times$ / ✓ / $\times$

Appendix

Table 10: Examples of model-generated explanations in our main evaluation. Color-highlighted extracts are discussed in Appendix I .

w_{con}

refers to the proportion of human annotators that had labeled the premise (P)-hypothesis (H) pair as a contradiction in Nie et al. (2020) . Token counts displayed combine both reasoning (where model is in thinking mode) and final output tokens. E: Efficiency, C: Coherence, S: Success, as judged by Gemma-4-31B-it.

Model	Explanation	E / C / S
Ex 4 ( $w_{con}$ : 0.95) \| P: All of them slept in one cave on animal skins, a single large clay pot cooked all of their food. H: They used multiple clay pots to cook their food.
5.6Sol-t GPT-5.6-sol thinking	Reasoning trace: I need to clarify […] premise mentions a single large pot cooking all food, but I wonder if multiple pots could have been used over time. […] The single cooking pot was replaced several times as it wore out , so multiple clay pots were used over time. (184 tokens)	✓ / ✓ / ✓
Q9B-n Qwen3.5-9B non-thinking	The single large clay pot mentioned was likely a communal vessel shared by the group, whereas individual pots were not used for cooking. (26 tokens)	$\times$ / ✓ / $\times$
Q9B-t Qwen3.5-9B thinking	Reasoning trace: […] 2. Identify the Conflict: - Premise: 1 pot. - Hypothesis: Multiple pots. […] 3. Brainstorm Reconciliations: - Idea 1: They started with one but broke it and got more? […] Better Angle: […]the hypothesis could refer to a broader collection of vessels they possessed […] Let’s stick with the temporal expansion. Draft: Their resourcefulness allowed them to replace the sole vessel with a set of multiple clay pots over time. […] Despite the initial reliance on a solitary vessel, the group eventually secured additional pots to manage their growing culinary requirements. (5251 tokens)	✓ / ✓ / ✓

Appendix

Table 11: Examples of LLM-generated explanations (continued from Table 10 ).

	Source Model of Generated Explanations	Coherence		Effectiveness		Explanation Success
	Source Model of Generated Explanations	Non- Thinking	Thinking	Non- Thinking	Thinking	Non- Thinking	Thinking
Qwen 3.5	Qwen3.5-0.8B	72.31 [-0.35ex][66.15, 78.46]	72.82 [-0.35ex][66.65, 79.49]	26.67 [-0.35ex][21.03, 32.82]	22.05 [-0.35ex][16.41, 27.18]	11.28 [-0.35ex][7.18, 15.90]	16.41 [-0.35ex][11.28, 21.03]
	Qwen3.5-2B	66.15 [-0.35ex][59.49, 72.32]	66.15 [-0.35ex][59.49, 73.33]	38.46 [-0.35ex][31.79, 45.13]	57.95 [-0.35ex][50.77, 64.10]	18.46 [-0.35ex][13.33, 24.10]	33.33 [-0.35ex][26.67, 39.49]
	Qwen3.5-4B	76.41 [-0.35ex][70.77, 81.54]	74.36 [-0.35ex][68.21, 80.51]	17.95 [-0.35ex][12.82, 23.08]	57.44 [-0.35ex][50.77, 64.12]	13.33 [-0.35ex][8.72, 17.95]	40.51 [-0.35ex][33.85, 47.69]
	Qwen3.5-9B	86.67 [-0.35ex][82.05, 91.28]	81.54 [-0.35ex][75.90, 86.67]	30.26 [-0.35ex][24.10, 37.44]	64.10 [-0.35ex][57.44, 70.77]	24.62 [-0.35ex][18.46, 30.77]	51.28 [-0.35ex][44.62, 57.95]
	Qwen3.5-27B	75.90 [-0.35ex][69.74, 81.54]	80.51 [-0.35ex][74.87, 86.15]	61.54 [-0.35ex][54.36, 68.21]	70.77 [-0.35ex][63.58, 76.94]	44.10 [-0.35ex][36.92, 51.28]	54.36 [-0.35ex][47.18, 61.03]

Appendix

Table 12: Extended table containing our main results in Table 2 , with 95% bootstrap confidence intervals enclosed in brackets. As before, coherence, effectiveness, and explanation success rate (in %) of each model’s generated explanations are as evaluated by Gemma-4-31B-it as the judge model (enclosed in blue ). Scores in the row corresponding to Gemma-4-31B-it are when the model acts as both explanation generator and judge. Highest value in each column is in bold .

	Source Model of Generated Explanations	Coherence		Effectiveness		Explanation Success
	Source Model of Generated Explanations	Non- Thinking	Thinking	Non- Thinking	Thinking	Non- Thinking	Thinking
Qwen 3.5	Qwen3.5-0.8B	60.68 (-11.63)	60.19 (-12.63)	26.21 (-0.46)	32.04 (+9.99)	10.19 (-1.09)	16.02 (-0.39)
	Qwen3.5-2B	57.77 (-8.38)	57.77 (-8.38)	43.69 (+5.23)	64.08 (+6.13)	18.45 (-0.01)	31.55 (-1.78)
	Qwen3.5-4B	67.96 (-8.45)	66.50 (-7.86)	25.24 (+7.29)	62.14 (+4.70)	14.08 (+0.75)	38.35 (-2.16)
	Qwen3.5-9B	79.61 (-7.06)	67.96 (-13.58)	34.95 (+4.69)	63.59 (-0.51)	23.30 (-1.32)	41.75 (-9.53)
	Qwen3.5-27B	70.39 (-5.51)	69.90 (-10.61)	67.96 (+6.42)	76.21 (+5.44)	45.15 (+1.05)	51.94 (-2.42)

Appendix

Table 13: Coherence, effectiveness, and success rate of each model’s generated explanations (in %), as evaluated by gpt-5.6-sol as the judge model (enclosed in blue ). Displayed delta figures ( + / - ) are computed against baseline results which use Gemma-4-31B-it as judge. Values outside the 95% bootstrap confidence intervals of the baseline results (see Table 12 ) are enclosed in red . Scores in the row corresponding to gpt-5.6-sol are when the model itself acts as both explanation generator and the judge doing the evaluating. Highest score in each column is in bold .

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Apr 15, 2026

Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis

Large language models (LLMs) have become increasingly used for various tasks, often coupled with Chain-of-Thought (CoT) prompting to boost accuracy. Recent work has shown that high label-prediction accuracy does not guarantee correct intermediate reasoning, and the causes of reasoning flaws vary from sample to sample, yet existing remedies either focus on a single domain or assume that one flaw type applies uniformly across samples. A simple mitigation method is to provide the model with the correct answer, but we show that this yields no consistent improvement in reasoning quality. This indicates that the problem cannot be fixed by LLMs' awareness of answers, and must instead be addressed through the structure of reasoning. Motivated by this, we propose CRAFT (Consensus Reasoning knowledge graph Aggregation for Flaw-aware Trace synthesis), which aggregates the consensus components shared across multiple candidate reasoning traces to synthesize improved ones. CRAFT consistently improves label-prediction accuracy on both logical and mathematical reasoning benchmarks, outperforming most baselines, while its post-processed traces achieve higher quality under fine-grained benchmark evaluation.

Zipeng Ling, Shuliang Liu, Seonil Son +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta · Alberta Machine Intelligence Institute +3

Reasoning Trace Analysis Logical Reasoning Self-Consistency Decoding CoT Reasoning Mathematical Reasoning Structured Reasoning+3

Ask PDF

Sep 15, 2026

An Empirical Study of Counterfactual Self-Explanations in LLMs

Large language models can easily generate explanations for their own outputs, but such self-explanations are not necessarily faithful to the model's behavior. We study this issue through counterfactual self-explanations, where a model minimally edits an input so that its own prediction changes. Across sentiment analysis and natural language inference, we evaluate ten instruction-tuned models from the LLaMA-3 and Qwen-2.5 families, measuring faithfulness, minimality, and alignment with human-annotated rationales. Our results show that model scale is the strongest determinant of explanation quality: larger models are substantially more likely to generate counterfactuals that flip their own predictions and target decision-relevant evidence. In contrast, the rationale-guided condition produces edit-minimal counterfactuals that are also more human-aligned. However, it does not consistently improve faithfulness. Overall, counterfactual self-explanations can provide useful behavioral evidence about model decisions, but their reliability depends strongly on model capacity and should be empirically validated rather than assumed.

Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis +2

National Technical University of Athens, Athens, Greece · Instituto de Telecomunicações, Lisbon, Portugal

Counterfactual Explanations Faithfulness of Language Model Explanations LLM Interpretability Counterfactual Reasoning in Language Models LLM Reliability

Ask PDF

Apr 16, 2026

Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoning

Large language models frequently produce mutually inconsistent answers when reasoning over multiple related queries. We study case-file logical consistency: maintaining a globally satisfiable belief state across interdependent queries. We introduce a benchmark of 390 multi-query reasoning instances with entailment/contradiction/unknown labels and propose set-level metrics including Case Satisfiability Rate, Contradiction Density and Revision Cost. Our solver-augmented approach extracts commitments, verifies global satisfiability and performs counterexample-guided repair. Across four reasoning domains, our method substantially reduces cross-query contradictions (SetCons: 0.56 to 0.94) while preserving per-query accuracy, demonstrating that global coherence is critical for robust multi-query reasoning.

Rohit Kumar Salla, Ramya Manasa Amancherla, Manoj Saravanan

Virginia Tech · Columbia University

LLM Reasoning

Ask PDF

Abstract

LLM Reasoning

Explore similar work

Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis

An Empirical Study of Counterfactual Self-Explanations in LLMs

Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoning