Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University
Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.
Figures & tables
Figure 1: (a) MAD relies on modality relevance alone and overlooks cross-modal conflicts. (b) CAD assesses conflict magnitude and actionability, then reallocates decoding weights to suppress unreliable fusion. (c) On CMM and AVHBench, CAD consistently outperforms Base, AVCD, and MAD with both Qwen2.5-Omni-7B and OmniVinci.
Figure 2: Fusion outcomes of Qwen2.5-Omni-7B on CMM. Axes show unimodal confidence in the ground-truth answer, and color denotes AV correctness. Highlighted examples show destructive fusion and complementary recovery.
Figure 3: Overview of CAD. Given an audio-visual input and textual query, CAD uses Potential Conflict Magnitude Estimation (PCME) to measure predictive disagreement across modality branches and Conflict Actionability Assessment (CAA) to assess whether the conflict is supported by reliable unimodal evidence. The actionable-conflict signal reallocates decoding influence from the joint audio-visual branch to modality-specific branches, yielding a response better grounded in the input evidence.
Model
CMM
AVHBench
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
VideoLLaMA2-AV
72.3
79.5
75.9
75.6
78.8
76.6
VideoLLaMA2-AV + AVCD
72.5
81.8
77.1
72.1
76.8
73.6
VideoLLaMA2-AV + MAD
81.8
82.3
82.0
78.5
80.4
79.7
VideoLLaMA2-AV + CAD
84.5
82.3
83.4
81.3
80.6
81.1
OmniVinci
82.3
85.5
83.9
74.5
71.2
73.4
Table 1: Main results on cross-modal hallucination benchmarks. We compare the original decoding strategy (Base), AVCD, MAD, and our proposed CAD. All results are reported in accuracy (%).
Fusion Method
AVHBench
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Baseline
75.7
76.9
76.1
Uniform
75.9
82.2
78.0
Argmax
78.6
81.8
79.6
CAD
83.2
85.9
84.1
Table 2: Comparison of different modality fusion strategies on AVHBench using Qwen2.5-Omni-7B. CAD achieves the highest overall accuracy among all strategies.
Decoding
Modality-Relevance Weights
AVHBench
Acc ↑
rA
rV
rAV
Video-Driven Audio Hall.
Audio-Driven Video Hall.
CAD
✓
✓
80.6
85.8
82.3
CAD
✓
✓
81.4
80.0
80.9
CAD
✓
✓
82.3
84.9
83.2
CAD
✓
✓
✓
83.2
85.9
84.1
Table 3: Ablation of modality-relevance weights in CAD on AVHBench using Qwen2.5-Omni-7B. We set each of rA , rV , and rAV to zero before coefficient construction. Using all three weights yields the highest overall accuracy.
Decoding
Conflict Assessment
AVHBench
Acc ↑
C
B
Video-Driven Audio Hall.
Audio-Driven Video Hall.
CAD
78.7
84.9
80.7
CAD
✓
81.5
85.0
82.7
CAD
✓
82.3
85.5
83.3
CAD
✓
✓
83.2
85.9
84.1
Table 4: Ablation of conflict-aware components in CAD on AVHBench using Qwen2.5-Omni-7B. C denotes potential cross-modal conflict magnitude, and B the reliability-weighted conflict-direction gate.
Figure 4: Distribution of potential conflict magnitude C for correct and wrong Qwen2.5-Omni-7B predictions on CMM.
Model
Worldsense
VideoMME
VideoLLaMA2-AV
25.9
47.9
VideoLLaMA2-AV + MAD
25.0
48.5
VideoLLaMA2-AV + CAD
28.8
49.4
Table 5: Comparison of Omni-LLMs on general AVQA benchmarks. All results are reported in accuracy (%).
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD
Sangyun Chung, Se Yeon Kim, Youngchae Chee +1
Integrated Vision Language Lab, KAIST, South Korea
Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as modality misrouting. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.
Yu Mi, Licheng Pan, Jiayi Zhu +7
Zhejiang University · Ant Group · Institute of Computing Technology, CAS
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.
Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani