Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning

Jing Tang; Kun Wang; Haolang Lu; Hongjin Chen; KaiTao Chen; Zhongxiang Sun; Qiankun Li; Lingjuan Lyu

cs.AIFeb 16, 2026

Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning

Authors: Jing Tang, Kun Wang, Haolang Lu, Hongjin Chen, KaiTao Chen, Zhongxiang Sun, Qiankun Li, Lingjuan Lyu,

Organizations: Huazhong University of Science and Technology · Nanyang Technological University · Beijing University of Posts and Telecommunications · Renmin University of China ·

Abstract

Multimodal large language models (MLLMs) in long chain-of-thought reasoning often fail when different knowledge sources provide conflicting signals. We formalize these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict. Through probing internal representations, we reveal that: (I) Linear Separability: different conflict types are explicitly encoded as linearly separable features rather than entangled; (II) Depth Localization: conflict signals concentrate in mid-to-late layers, indicating a distinct processing stage for conflict encoding; (III) Hierarchical Consistency: aggregating noisy token-level signals along trajectories robustly recovers input-level conflict types; and (IV) Directional Asymmetry: reinforcing the model's implicit source preference under conflict is far easier than enforcing the opposite source. Our findings provide a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.

Figures & tables

Figure 1: Overview of Knowledge Sources and Conflict Types. We categorize knowledge into Visual ( $\mathcal{K}_{\text{vision}}$ ), Textual ( $\mathcal{K}_{\text{text}}$ ), and Parametric Prior ( $\mathcal{K}_{\text{prior}}$ ). Knowledge conflicts arise when factual statements from different sources act as incompatible signals. We define three primary conflict types: Vision-Text ( $\mathcal{C}_{\mathrm{VT}}$ ), Vision-Prior ( $\mathcal{C}_{\mathrm{VP}}$ ), and Prior-Text ( $\mathcal{C}_{\mathrm{PT}}$ ).

Metric	Llama-3.2V-11B-cot				R1-Onevision-7B				Ocean-R1-7B-Instruct
Metric	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	All	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	All	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	All
Samples	749	1012	803	2564	724	993	769	2486	640	1026	807	2473
Avg. CoT length (tokens)	326.79	1768.85	238.50	868.32	706.85	790.63	558.97	694.57	488.15	711.26	302.97	520.28
Avg. conflict spans per sample	2.69	6.20	4.04	4.50	3.66	6.73	7.02	5.93	8.68	9.00	5.43	7.75
Conflict token density (%)	4.92	1.65	11.25	5.61	3.20	2.16	7.68	4.17	8.70	3.23	11.77	7.43
Conflict Sample Ratio (%)	63.68	82.21	86.43	78.12	59.67	85.90	87.91	78.88	88.75	90.25	89.34	89.57

Table 1: Output-level conflict profile across models (objective conflict subsets). We present statistics of generated trajectories under three types of conflict (model details in Appendix B ). Metrics reported include sample count, average CoT length, average conflict spans per sample ( spans are contiguous conflict segments identified via an automated LLM annotation pipeline, may consist of one or multiple tokens. ), conflict token density (proportion of conflicting tokens), and sample conflict rate (% of samples exhibiting effective conflict ).

Figure 2: Token-level separability of effective conflict $\mathcal{C}^{e}_{i,j}(t\mid x)$ . The left panel shows the confusion matrix over token-level conflict predictions. The right panels decompose performance into binary detection of conflict versus no-conflict, and fine-grained attribution among conflict types. Values denote row-normalized recall.

Figure 3: Sample-level separability of conflict types. We visualize the t-SNE projection of hidden states at layer 20 ( R1-Onevision ) and layer 39 ( Llama-3.2V ). The three conflict categories are colored according to their Objective Conflict labels, pre-defined during dataset construction. The top-right confusion matrices illustrate the sample-level attribution performance.

Figure 4: Cross-layer distribution of conflict signals. Top row: attention-head activation ratio on conflict tokens vs. no-conflict tokens (lines), and their difference (bars), computed using effective conflict labels. Middle/bottom rows: layer-wise probe performance (one-vs-rest AUC and Recall@0.1) for $\mathcal{C}_{\mathrm{VP}},\mathcal{C}_{\mathrm{PT}},\mathcal{C}_{\mathrm{VT}}$ across three MLLM backbones. — Figure 4: Cross-layer distribution of conflict signals. Top row: attention-head activation ratio on conflict tokens vs. no-conflict tokens (lines), and their difference (bars), computed using effective conflict labels. Middle/bottom rows: layer-wise probe performance (one-vs-rest AUC and [email protected]) for $\mathcal{C}_{\mathrm{VP}},\mathcal{C}_{\mathrm{PT}},\mathcal{C}_{\mathrm{VT}}$ across three MLLM backbones.

Models	Probe	Granularity	AUC (%)				[email protected] (%)
Models	Probe	Granularity	w/o Conflict	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	w/o Conflict	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	$\mathcal{C}^{o}_{\mathrm{VT}}=1$
R1-Onevision (7B)	Linear	All Token	81.7 ±0.1	86.3 ±0.2	92.0 ±0.1	94.8 ±0.2	50.0 ±0.3	56.8 ±0.2	75.1 ±0.1	87.3 ±0.3
		Span Only	76.8 ±0.2	82.5 ±0.1	90.8 ±0.3	95.4 ±0.2	35.5 ±0.2	44.5 ±0.3	70.5 ±0.2	88.5 ±0.1
		Span-Max	93.2 ±0.1	94.2 ±0.2	98.6 ±0.1	97.3 ±0.1	81.5 ±0.2	82.4 ±0.1	97.2 ±0.1	93.8 ±0.2
	MLP	All Token	95.5 ±0.1	90.4 ±0.2	85.2 ±0.3	94.1 ±0.1	89.1 ±0.2	68.4 ±0.1	62.7 ±0.2	79.3 ±0.1
		Span Only	95.7 ±0.2	86.1 ±0.3	80.3 ±0.1	93.3 ±0.2	89.8 ±0.1	53.0 ±0.2	43.7 ±0.2	76.8 ±0.3

Table 2: Assessment of conflict probe performance across three VLM backbones. We report AUC and Recall at FPR=0.1 ([email protected]) under the One-vs-Rest setting. Gray rows indicate the Span-Max aggregation, which consistently outperforms token-level baselines. Values are presented as percentages (%).

Figure 5: Semantic performance of targeted source control. We evaluate three conflict subsets ( $\mathcal{C}^{o}_{\mathrm{VP}},\mathcal{C}^{o}_{\mathrm{VT}},\mathcal{C}^{o}_{\mathrm{PT}}$ ) using judge-based metrics: ASR (Anchor Support Rate, $\uparrow$ ), ARR (Anchor Rejection Rate, $\downarrow$ ), and OER (Obvious Error Rate, $\downarrow$ ). Forward/Reverse denote intervening toward the truth-anchored (benchmark-reliable) vs. conflicting source. Arrows indicate relative changes against the baseline. Note that VCD is inapplicable to the non-visual $\mathcal{C}^{o}_{\mathrm{PT}}$ subset.

	Subset	R1-Onevision-7B				Ocean-R1-7B-Instruct				Llama-3.2V-11B-cot
		SS $\uparrow$	CAC $\downarrow$	CCI $\downarrow$	CR $\downarrow$	SS $\uparrow$	CAC $\downarrow$	CCI $\downarrow$	CR $\downarrow$	SS $\uparrow$	CAC $\downarrow$	CCI $\downarrow$	CR $\downarrow$
	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	0.94	0.04	0.70	0.03	0.89	0.07	0.71	0.06	0.94	0.04	0.45	0.02
	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	0.88	0.08	0.80	0.10	0.87	0.09	0.79	0.10	0.90	0.06	0.72	0.03
baseline	$\mathcal{C}^{o}_{\mathrm{PT}}=1$	0.82	0.12	0.80	0.15	0.82	0.12	0.80	0.15	0.84	0.11	0.70	0.11
VCD	$\mathcal{C}^{o}_{\mathrm{VP}}=1$	0.92 -0.02	0.05 +0.01	0.69 -0.00	0.04 +0.01	0.90 +0.01	0.06 -0.01	0.69 -0.01	0.05 -0.01	0.85 -0.09	0.08 +0.04	0.63 +0.18	0.06 +0.05
VCD	$\mathcal{C}^{o}_{\mathrm{VT}}=1$	0.90 +0.01	0.07 -0.01	0.79 -0.01	0.08 -0.01	0.92 +0.05	0.05 -0.03	0.75 -0.04	0.05 -0.05	0.78 -0.12	0.12 +0.06	0.69 -0.03	0.15 +0.11

Table 3: Token-level conflict mitigation under the forward direction. Results are reported on three objective-conflict subsets (

\mathcal{C}^{o}_{\mathrm{VP}}=1

\mathcal{C}^{o}_{\mathrm{VT}}=1

, and

\mathcal{C}^{o}_{\mathrm{PT}}=1

) across three backbones. We report four token-level mitigation metrics: SS

\uparrow

, CR

\downarrow

, CAC

\downarrow

and CCI

\downarrow

(metric definitions in Appendix D.3 ). VCD is not applicable when

\mathcal{C}^{o}_{\mathrm{PT}}=1

and is therefore reported only for the first two subsets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model Backbone	Split	Samples	Total Spans	Avg. Spans/Sample
R1-Onevision-7B	Test	498	9,038	18.1
R1-Onevision-7B	Train	1,987	34,300	17.3
Ocean-R1-7B	Test	495	12,881	26.0
Ocean-R1-7B	Train	1,978	51,931	26.3
Llama-3.2V-11B-cot	Test	513	9,419	18.4
Llama-3.2V-11B-cot	Train	2,051	37,118	18.1

Appendix

Table 4: Statistics of the Objective Conflict Dataset and Generated Trajectories. The dataset provides span-level supervision for training conflict probes (Section 4 ) and evaluating interventions (Section 5 ).

Attribute	R1-Onevision	Ocean-R1	Llama-3.2V-cot
Params	7B	7B	11B
Layers	28	28	40
Hidden Dim	3584	3584	4096
Conflict Layer	20	20	39
Base Model	Qwen2.5-VL	Qwen2.5-VL	Llama-3.2-Vision

Appendix

Table 5: Summary of evaluated MLLM backbones. The “Conflict Layer” indicates the layer used for probe training and intervention (identified in Section 4.3 ).

Model	Linear AUC	Peak Layer	$\mathcal{C}_{\mathrm{PT}}$ AUC	$\mathcal{C}_{\mathrm{VP}}$ AUC	CR Red.
R1-Onevision	96.2%	L20 (71%)	98.6%	93.4%	80%
Ocean-R1	95.8%	L20 (71%)	98.6%	93.2%	76%
Llama-3.2V	97.1%	L39 (78%)	99.2%	95.9%	67%

Appendix

Table 6: Cross-model consistency of conflict encoding patterns. Despite architectural differences, all three backbones exhibit similar separability, depth localization, and intervention response.

Type	Conflicting Sources	Forward ( $\uparrow$ )	Reverse ( $\downarrow$ )
$\mathcal{C}_{\mathrm{VP}}$	Vision vs. Prior	$\mathcal{K}_{\text{vision}}$	$\mathcal{K}_{\text{prior}}$
$\mathcal{C}_{\mathrm{VT}}$	Vision vs. Text	$\mathcal{K}_{\text{vision}}$	$\mathcal{K}_{\text{text}}$
$\mathcal{C}_{\mathrm{PT}}$	Prior vs. Text	$\mathcal{K}_{\text{prior}}$	$\mathcal{K}_{\text{text}}$

Appendix

Table 7: Forward and Reverse direction mappings for each conflict type. Forward aligns with the more reliable source based on our data construction. Reverse enforces the opposite choice.

Metric	Direction	Interpretation
SS (Support Score)	$\uparrow$	Avg. probability of no-conflict state
CR (Conflict Rate)	$\downarrow$	Fraction of tokens with predicted conflict
CAC (Confidence-Adjusted)	$\downarrow$	Penalizes confident conflict predictions
CCI (Conflict Confidence)	$\downarrow$	Avg. confidence when in conflict state

Appendix

Table 8: Summary of token-level probe metrics used for conflict mitigation evaluation.

Hyperparameter	R1-Onevision	Ocean-R1	Llama-3.2V
Layer Selection
Intervention layer $l$	20	20	39
VCD Parameters
Correction strength $\beta$	0.8	0.8	0.8
Contrastive input	Blank image	Blank image	Blank image
Steering Parameters

Appendix

Table 9: Intervention hyperparameters for each backbone. These settings are used in Section 5 for both targeted source control (Figure 5 ) and conflict mitigation (Table 3 ).

Knowledge Conflicts in Language Models

Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.

See all 142 papers

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
Sep 1, 2026Jungyeon Lee, Yejin Yoon, Taeuk KimMultimodal Robustness Knowledge Conflicts in Language Models
Hanyang University, Seoul, Republic of Korea
Large Language Models in Resolving Contextual Knowledge Conflicts
Sep 2, 2026Xinye Yang, Zhenyang Liu, Ruisi Li +1Knowledge Conflicts in Language Models LLM Interpretability
1Northwestern University, Evanston, IL · 2New York University, New York, NY · 3Computer & Information Science and Engineering, University of Florida, Gainesville, FL
Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts
Aug 2, 2025Xianda Zheng, Zijian Huang, Meng-Fen Chiang +4Knowledge Conflicts in Language Models
School of Computer Science, University of Auckland · Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University · School of Computing and Information Systems, Singapore Management University +1

Sep 1, 2026J/K move · Enter open · S save

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

Jungyeon Lee, Yejin Yoon, Taeuk Kim

Hanyang University, Seoul, Republic of Korea

Multimodal Robustness Knowledge Conflicts in Language Models Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered image of the same passage, or as both together. However, it remains unclear how consistently these surface forms are processed, especially when the evidence conflicts with the model's parametric knowledge. We study modality robustness under knowledge conflict across 13 MLLMs and two datasets, and find them far from robust. (1) Contrary to common belief, models favor a context that contradicts parametric knowledge more readily in image form than in text form; (2) when a contradicting text and image are presented together, the preferred modality is essentially arbitrary, varying with input order, model, and dataset. We further demonstrate that this instability has practical consequences: it degrades performance in multimodal RAG and can be exploited by adversarial attacks. To alleviate this brittleness, we examine several simple techniques---prompting, steering, supervised fine-tuning (SFT), and direct preference optimization; the majority prove ineffective, whereas SFT achieves moderate success. We therefore call for greater awareness of this inconsistency and argue that it is fundamental, demanding attention at multiple training stages.

Abstract

Knowledge Conflicts in Language Models

Explore similar work

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

Large Language Models in Resolving Contextual Knowledge Conflicts

Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts