Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method

Tianzhe Zhao; Jiaoyan Chen; Shuxiu Zhang; Haiping Zhu; Qika Lin; Jun Liu

cs.CLApr 13, 2026

Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method

Authors: Tianzhe Zhao, Jiaoyan Chen, Shuxiu Zhang, Haiping Zhu, Qika Lin, Jun Liu

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University · Department of Computer Science, The University of Manchester · Hunan University · National University of Singapore

Abstract

Large language models (LLMs) have achieved remarkable success across a wide range of applications especially when augmented by external knowledge through retrieval-augmented generation (RAG). Despite their widespread adoption, recent studies have shown that LLMs often struggle to perform faithful reasoning when conflicting knowledge is retrieved. However, existing work primarily focuses on conflicts between external knowledge and the parametric knowledge of LLMs, leaving conflicts across external knowledge largely unexplored. Meanwhile, modern RAG systems increasingly emphasize the integration of unstructured text and (semi-)structured data like knowledge graphs (KGs) to improve knowledge completeness and reasoning faithfulness. To address this gap, we introduce ConflictQA, a novel benchmark that systematically instantiates conflicts between textual evidence and KG evidence. Extensive evaluations across representative LLMs reveal that, facing such cross-source conflicts, LLMs often fail to identify reliable evidence for correct reasoning. Instead, LLMs become more sensitive to prompting choices and tend to rely exclusively on either KG or textual evidence, resulting in incorrect responses. Based on these findings, we further propose XoT, a two-stage explanation-based thinking framework tailored for reasoning over heterogeneous conflicting evidence, and verify its effectiveness with extensive experiments.

Figures & tables

Figure 1. Demonstration of cross-source knowledge conflict in LLM reasoning with RAG.

Figure 2. The pipeline for constructing the ConflictQA benchmark, mainly including positive and conflicting evidence construction as well as examples under both Non-COMP and COMP settings.

Conflict	#Avg Triples	#Avg Tokens	#Sample
Non-Complementary
Positive	1.84	390.41	802
TripleConf	1.84	390.41	802
TextConf	1.84	330.82	802
Complementary
TextPos	1.05	534.92	430

Table 1. Statistics of ConflictQA.

Large Language Model	TripleConf						TextConf
	F1			EM			F1			EM
	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$
General LLMs
Qwen3-8B	86.01	21.85	$64.16_{75\%\downarrow}$	76.28	7.61	$68.67_{90\%\downarrow}$	86.01	76.24	$9.77_{11\%\downarrow}$	76.28	52.44	$23.84_{31\%\downarrow}$
Llama-3.1-8B-Instruct	67.30	36.82	$30.48_{45\%\downarrow}$	50.75	19.20	$31.55_{62\%\downarrow}$	67.30	63.39	$3.91_{6\%\downarrow}$	50.75	41.15	$9.60_{19\%\downarrow}$
Llama-3.1-70B-Instruct	84.72	46.83	$37.89_{45\%\downarrow}$	79.80	32.67	$47.13_{59\%\downarrow}$	84.72	63.10	$21.62_{26\%\downarrow}$	79.80	48.96	$30.84_{39\%\downarrow}$

Table 2. Performance (%) of LLMs on ConflictQA under the non-complementary setting. Pos and Conf denote performance with positive and conflicting evidence, respectively.

\Delta

indicates the resulting performance degradation. Bold and underline values denote optimal and sub-optimal results, respectively. Pink cells highlight cases with substantial degradation.

Large Language Model	TripleConf						TextConf
	F1			EM			F1			EM
	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$	Pos	Conf	$\Delta$
General LLMs
Qwen3-8B	53.26	26.26	$27.00_{51\%\downarrow}$	36.98	8.14	$28.84_{78\%\downarrow}$	88.99	68.73	$20.26_{23\%\downarrow}$	78.63	46.98	$31.65_{40\%\downarrow}$
Llama-3.1-8B-Instruct	52.79	36.45	$16.34_{31\%\downarrow}$	38.37	17.67	$20.70_{54\%\downarrow}$	80.39	42.56	$37.83_{47\%\downarrow}$	64.19	21.63	$42.56_{66\%\downarrow}$
Llama-3.1-70B-Instruct	57.76	42.14	$15.62_{27\%\downarrow}$	46.74	31.16	$15.58_{33\%\downarrow}$	83.00	47.46	$35.54_{43\%\downarrow}$	74.65	33.72	$40.93_{55\%\downarrow}$

Table 3. Performance (%) under the complementary setting. Its settings are consistent with Table 2.

Large Language Model	TripleConf		TextConf
Large Language Model	$\Delta$ F1	$\Delta$ EM	$\Delta$ F1	$\Delta$ EM
Qwen3-8B	19.74	8.73	-0.30	-0.87
Llama-3.1-8B-Instruct	-4.72	4.74	-0.33	-9.48
Llama-3.1-70B-Instruct	-1.41	-0.50	-4.42	-6.86
Open-Mistral-7B	9.79	8.48	7.87	8.10
Mistral-Large-2512	1.93	6.98	5.12	-1.74

Table 4. Performance gap caused by evidence ordering in the non-complementary setting. Light-blue cells indicate negative gaps.

Figure 3. Exact Match results (%) of LLMs utilizing different prompts under non-complementary reasoning scenarios.

Figure 4. Exact Match results (%) of LLMs utilizing different prompts under complementary reasoning scenarios.

Large Language Models		Non-COMP				COMP
		TripleConf		TextConf		TripleConf		TextConf
		F1	EM	F1	EM	F1	EM	F1	EM
Qwen3-8B	XoT	46.07	13.34	55.35	21.57	45.10	15.58	61.06	23.72
Qwen3-8B	$\Delta$	24.22 111% $\uparrow$	5.73 75% $\uparrow$	20.89 27% $\downarrow$	30.87 59% $\downarrow$	18.87 72% $\uparrow$	7.44 91% $\uparrow$	7.67 11% $\downarrow$	23.26 50% $\downarrow$
Open-Mistral-7B	XoT	36.88	9.35	48.01	18.45	37.80	13.02	51.02	23.26
Open-Mistral-7B	$\Delta$	7.19 24% $\uparrow$	4.12 31% $\downarrow$	19.09 28% $\downarrow$	28.62 61% $\downarrow$	24.46 183% $\uparrow$	6.51 100% $\uparrow$	0.79 2% $\downarrow$	14.88 39% $\downarrow$

Table 5. Performance (%) of XoT on the ConflictQA benchmark. Bold and underline values denote optimal and sub-optimal results, respectively. Cells in pink indicate performance improvements of XoT over the conflict-aware QA prompt.

Figure 5. An example illustrating the predictions of GPT-4o under different reasoning strategies on ConflictQA in the COMP setting with misleading textual evidence. Entities highlighted in green and pink denote the golden answer and the misleading answer, respectively.

Knowledge Conflicts in Language Models

Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.

See all 142 papers

Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts
Aug 2, 2025Xianda Zheng, Zijian Huang, Meng-Fen Chiang +4Knowledge Conflicts in Language Models
School of Computer Science, University of Auckland · Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University · School of Computing and Information Systems, Singapore Management University +1
Large Language Models in Resolving Contextual Knowledge Conflicts
Sep 2, 2026Xinye Yang, Zhenyang Liu, Ruisi Li +1Knowledge Conflicts in Language Models LLM Interpretability
1Northwestern University, Evanston, IL · 2New York University, New York, NY · 3Computer & Information Science and Engineering, University of Florida, Gainesville, FL
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Oct 18, 2026Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1Retrieval-Augmented Reasoning LLM Reasoning
Aarhus University · Microsoft Research · Technische Universität Darmstadt

Aug 2, 2025J/K move · Enter open · S save

Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts

Xianda Zheng, Zijian Huang, Meng-Fen Chiang, Jiamou Liu+3

School of Computer Science, University of Auckland · Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University · School of Computing and Information Systems, Singapore Management University+1

Knowledge Conflicts in Language Models

Open paper AskPDF arXiv

Explicit knowledge conflicts, occurring when retrieved contexts contain contradictory information, pose a fundamental challenge for Large Language Models (LLMs) as they integrate increasingly diverse data sources. The core difficulty lies in the complexity of entangled narratives and heterogeneous conflict patterns, which frequently exceeds the reasoning capacity of standard backbone architectures. We propose Kcr (Knowledge Conflict Reasoning), a framework that adjudicates contradictions by systematically structuring their underlying logic. Kcr disentangles conflicting contexts into discrete sets of reasoning traces, utilizing a hybrid representation of text and graphs to facilitate systematic comprehension. It then employs a Reinforcement Learning with Verifiable Rewards (RLVR) paradigm to instill a reasoning policy that maximizes logical consistency while suppressing spurious paths derived from contradictory evidence. Extensive evaluations demonstrate that Kcr yields substantial performance gains. Notably, a 7B model enhanced by Kcr achieves adjudication capabilities that significantly outperform leading proprietary models, including GPT-4o and GPT-5.1, on complex tasks. Code is available at https://github.com/zhengxianda/KCR.

Abstract

Knowledge Conflicts in Language Models

Explore similar work

Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts

Large Language Models in Resolving Contextual Knowledge Conflicts

Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge