CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Yuchen Deng; Chang Sun; Hai-Tao Zheng; Feidiao Yang; Yuxing Han

eess.ASAug 26, 2026

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Authors: Yuchen Deng, Chang Sun, Hai-Tao Zheng, Feidiao Yang, Yuxing Han

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University

Abstract

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.

Figures & tables

Figure 1: (a) MAD relies on modality relevance alone and overlooks cross-modal conflicts. (b) CAD assesses conflict magnitude and actionability, then reallocates decoding weights to suppress unreliable fusion. (c) On CMM and AVHBench, CAD consistently outperforms Base, AVCD, and MAD with both Qwen2.5-Omni-7B and OmniVinci.

Figure 2: Fusion outcomes of Qwen2.5-Omni-7B on CMM. Axes show unimodal confidence in the ground-truth answer, and color denotes $AV$ correctness. Highlighted examples show destructive fusion and complementary recovery.

Figure 3: Overview of CAD. Given an audio-visual input and textual query, CAD uses Potential Conflict Magnitude Estimation (PCME) to measure predictive disagreement across modality branches and Conflict Actionability Assessment (CAA) to assess whether the conflict is supported by reliable unimodal evidence. The actionable-conflict signal reallocates decoding influence from the joint audio-visual branch to modality-specific branches, yielding a response better grounded in the input evidence.

Model	CMM			AVHBench
	Visual Dom.	Audio Dom.	Overall Acc.	Video-Driven Audio Hall.	Audio-Driven Video Hall.	Overall Acc.
VideoLLaMA2-AV	72.3	79.5	75.9	75.6	78.8	76.6
VideoLLaMA2-AV + AVCD	72.5	81.8	77.1	72.1	76.8	73.6
VideoLLaMA2-AV + MAD	81.8	82.3	82.0	78.5	80.4	79.7
VideoLLaMA2-AV + CAD	84.5	82.3	83.4	81.3	80.6	81.1
OmniVinci	82.3	85.5	83.9	74.5	71.2	73.4

Table 1: Main results on cross-modal hallucination benchmarks. We compare the original decoding strategy (Base), AVCD, MAD, and our proposed CAD. All results are reported in accuracy (%).

Fusion Method	AVHBench
Fusion Method	Video-Driven Audio Hall.	Audio-Driven Video Hall.	Overall Acc.
Baseline	75.7	76.9	76.1
Uniform	75.9	82.2	78.0
Argmax	78.6	81.8	79.6
CAD	83.2	85.9	84.1

Table 2: Comparison of different modality fusion strategies on AVHBench using Qwen2.5-Omni-7B. CAD achieves the highest overall accuracy among all strategies.

Decoding	Modality-Relevance Weights			AVHBench		Acc $\uparrow$
Decoding	$r_{A}$	$r_{V}$	$r_{AV}$	Video-Driven Audio Hall.	Audio-Driven Video Hall.	Acc $\uparrow$
CAD		✓	✓	80.6	85.8	82.3
CAD	✓		✓	81.4	80.0	80.9
CAD	✓	✓		82.3	84.9	83.2
CAD	✓	✓	✓	83.2	85.9	84.1

Table 3: Ablation of modality-relevance weights in CAD on AVHBench using Qwen2.5-Omni-7B. We set each of

r_{A}

r_{V}

, and

r_{AV}

to zero before coefficient construction. Using all three weights yields the highest overall accuracy.

Decoding	Conflict Assessment		AVHBench		Acc $\uparrow$
Decoding	$C$	$B$	Video-Driven Audio Hall.	Audio-Driven Video Hall.	Acc $\uparrow$
CAD			78.7	84.9	80.7
CAD	✓		81.5	85.0	82.7
CAD		✓	82.3	85.5	83.3
CAD	✓	✓	83.2	85.9	84.1

Table 4: Ablation of conflict-aware components in CAD on AVHBench using Qwen2.5-Omni-7B.

C

denotes potential cross-modal conflict magnitude, and

B

the reliability-weighted conflict-direction gate.

Figure 4: Distribution of potential conflict magnitude $C$ for correct and wrong Qwen2.5-Omni-7B predictions on CMM.

Model	Worldsense	VideoMME
VideoLLaMA2-AV	25.9	47.9
VideoLLaMA2-AV + MAD	25.0	48.5
VideoLLaMA2-AV + CAD	28.8	49.4

Table 5: Comparison of Omni-LLMs on general AVQA benchmarks. All results are reported in accuracy (%).

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,085 papers

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Jan 29, 2026Sangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs
Oct 13, 2026Yu Mi, Licheng Pan, Jiayi Zhu +7Multimodal Large Language Models Multimodal Hallucination
Zhejiang University · Ant Group · Institute of Computing Technology, CAS
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
Mar 3, 2026Ashutosh Chaubey, Jiacheng Pang, Mohammad SoleymaniMultimodal Large Language Models Multimodal Hallucination
University of Southern California

Jan 29, 2026J/K move · Enter open · S save

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro

Integrated Vision Language Lab, KAIST, South Korea

Multimodal Robustness Hallucination in Language Models Multimodal Large Language Models Multimodal Hallucination Contrastive Decoding Multimodal Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD

Abstract

Multimodal Large Language Models

Explore similar work

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization