MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

Ashutosh Chaubey; Jiacheng Pang; Mohammad Soleymani

cs.CVMar 3, 2026

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

Authors: Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani

Organizations: University of Southern California

Abstract

Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.

Figures & tables

Figure 1: Comparison of the proposed MoD-DPO with other preference optimization baselines for mitigating cross-modal hallucinations arising from spurious inter-modality correlations and over-reliance on language priors.

Figure 2: Comparison of proposed MoD-DPO++ with other preference optimization baselines on AVHBench [ 36 ] and CMM [ 26 ] . Average accuracy is reported. ADVH: Audio-driven video hallucination, VDAH: Video-driven audio hallucination.

Figure 3: Modality Decoupled Preference Optimization. In addition to the response preference and reference regularization terms in DPO [ 32 ] , we include additional KL regularization terms to increase model invariance to irrelevant modalities and model sensitivity to relevant modalities. Additionally, we penalize response generation with only text inputs to remove language priors in the model.

Figure 4: Preference Data Generation Pipeline. We disentangle the audiovisual input to obtain separate audio and visual captions or tags (Stage 1), which are then used to generate QA pairs for Stage 2. Finally, we create preference data for modality-specific questions by constructing chosen responses using relevant modality information and rejected responses using irrelevant modality information.

Panel of Figure 5 — Figure 5: Preference Data Statistics. (Left) Number of preference samples generated from different source datasets. (Right) Composition of samples belonging to different tasks.

	Audio-driven Video Hallucination				Video-driven Audio Hallucination				Audiovisual Matching
Method	Acc.	Pre.	Rec.	F1	Acc.	Pre.	Rec.	F1	Acc.	Pre.	Rec.	F1
VideoLLaMA 2 [ 9 ]	79.23	81.51	76.94	79.16	75.07	85.15	64.98	73.71	52.93	92.11	13.75	23.93
VITA-1.5 [ 11 ]	67.17	78.17	56.17	65.36	54.01	45.85	62.18	52.78	46.85	66.31	27.40	38.78
OmniVinci [ 48 ]	61.36	57.92	64.79	61.16	58.56	84.71	32.40	46.88	54.32	87.42	21.21	34.14
Qwen 3 Omni [ 45 ]	83.54	85.92	81.17	83.47	76.46	94.94	57.99	72.00	58.52	97.76	19.14	32.02
Qwen 2.5 Omni [ 44 ]	84.15	76.76	91.55	83.51	77.38	94.93	59.83	73.39	54.69	9.81	99.57	17.85

Table 1: Comparison of the proposed approach with other DPO and omni LLM methods on AVHBench [ 36 ] .

	Spurious Inter-modality Correlation						Uni-modality Over-reliance						Overall
	VL		AL		VAL		Visual Dom.		Audio Dom.		Lang. Dom.
Method	pa	hr	pa	hr	pa	hr	pa	hr	pa	hr	pa	hr	pa	hr
VideoLLaMA 2 [ 9 ]	75.0	86.0	77.5	94.0	78.0	98.0	62.0	75.5	80.0	90.0	57.5	43.0	71.7	81.1
VITA 1.5 [ 11 ]	92.5	91.5	42.5	30.5	77.7	72.9	57.0	24.4	77.4	67.7	86.0	55.5	72.2	57.1
OmniVinci [ 48 ]	88.5	81.0	74.5	90.0	98.0	72.0	91.0	63.5	96.0	35.0	87.0	75.0	89.2	69.4
Qwen 3 Omni [ 45 ]	96.5	77.5	93.0	88.0	97.5	94.0	95.5	60.0	94.5	67.0	93.0	65.0	95.0	75.3

Table 2: Comparison of the proposed approach with other DPO and omni LLM methods on Curse of Multi-Modalities [ 26 ] .

			AVHBench [ 36 ]			CMM [ 26 ]
Sens.	Inv.	LPD	Acc.	Pre.	Rec.	pa	hr
✗	✗	✗	72.07	60.50	83.65	86.4	84.6
✓	✗	✗	77.02	69.82	84.21	87.4	85.1
✗	✓	✗	76.04	67.18	84.89	87.0	85.4
✓	✓	✗	79.74	73.74	85.72	88.8	86.2
✗	✗	✓	73.88	63.18	84.58	86.9	86.3

Table 3: Ablation study shows performance improvement of Qwen 2.5 Omni [ 44 ] using different components of the proposed MoD-DPO++. LPD: Language Prior Debiasing.

			AVHBench [ 36 ]			CMM [ 26 ]
$y_{l}$	$a^{\prime}$ / $v^{\prime}$	$t$	Acc.	Pre.	Rec.	pa	hr
Rel. Opp.	O	-	77.03	69.14	84.91	87.0	86.1
Rel. Opp.	RN	-	79.93	73.98	85.87	88.8	87.3
Rel. Opp.	RAV	-	79.03	71.87	86.18	88.4	86.3
Rel. Opp.	Diff.	10	72.72	62.16	83.28	86.3	84.9
Rel. Opp.	Diff.	50	76.92	69.52	84.32	87.1	85.4

Table 4: Performance comparison using different forms of rejected samples for MoD-DPO++ on Qwen 2.5 Omni [ 44 ] .

y_{l}

: rejected text response,

a^{\prime}

v^{\prime}

: corrupt audio/visual inputs, O: all zeros, RN: gaussian random noise, RAV: random audio/visual input, Diff.: diffusion-based corruption,

t

: steps for diffusion corruption.

Panel of Figure 7 — Figure 7: Distribution shift for the log-likelihood of the correct answer when the relevant modality is corrupted ( left two plots ) and when the irrelevant modality is corrupted ( right two plots ) for the audio-driven visual hallucination task in AVHBench [ 36 ] . MoD-DPO++ results in a larger shift when the relevant modality is corrupted ( sensitivity ) and lower shift when the irrelevant modality is corrupted ( invariance ).

		AVHBench [ 36 ]			CMM [ 26 ]
AV Context	Tasks	Acc.	Pre.	Rec.	pa	hr
Mat.	A,V	76.69	69.29	84.09	87.6	85.8
Mis.	A,V	78.24	72.18	84.29	88.2	86.1
Mat.+Mis.	A,V	80.42	74.50	86.34	89.2	87.2
Mat.+Mis.	A,V, AV	80.33	74.68	85.98	89.0	87.5

Table 6: Effect of training with matched/mismatched audiovisual contexts and tasks in the preference training data. Reference is Qwen 2.5 Omni [ 44 ] . A: Audio-related, V: Video-related, AV: Joint audiovisual tasks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Audiovisual Input	Prompt $x$	Chosen Response $y_{w}$	Rejected Response $y_{l}$
Audio: “(machine whirring…)”	Is the sewing machine visible in the video?	Yes, the object is visible in the video. The video captures a close-up view of an embroidery machine in action, stitching a logo onto a cap.	No, the object is not visible in the video. We can only hear the sound of a machine operating with some noise.
Audio: “(dog panting…)”	Is the chair making sound in the audio?	No, the object is not making a sound in the audio. The audio indicates that the primary sound is panting, likely from a dog, and there is no sound coming from a chair.	Yes, the object is making a sound in the audio. The dog can be seen barking on the noisy chair.

Appendix

Table 8: Examples from the generated preference dataset.

Figure 9: Effect of varying different hyperparameters involved in MoD-DPO. Default values: $\beta_{\text{sens}}=0.05$ , $\beta_{\text{inv}}=0.02$ , $\gamma_{\text{LPD}}=0.05$ . When varying one hyperparameter, others are set to their default values.

	AVHBench [ 36 ]			CMM [ 26 ]
Method	Acc.	Pre.	Rec.	pa	hr
Qwen 2.5 Omni [ 44 ]	72.07	60.50	83.65	86.4	84.6
+ VCD* [ 22 ]	74.74	61.64	87.84	86.9	85.2
+ AVCD [ 22 ]	75.57	62.85	88.28	87.2	85.6
+ MoD-DPO++	80.42	74.50	86.34	89.2	87.2

Appendix

Table 9: Comparison with decode-time approaches.

Method	OmniDialog
Qwen 2.5 Omni	83.91
+ OmniDPO	84.18
+ MoD-DPO	83.96
+ MoD-DPO++	85.86

Appendix

Table 10: OmniDialog Results

Figure 10: Audio and visual captioning prompts used in Stage-1 of Preference Data Generation Sec. 4.3 . Audio captions are obtained from AudioFlamingo 3 [ 13 ] and visual captions are obtained from GPT-4o [ 19 ] .

Figure 11: QA Generation Prompt - Stage 2.1. used for classifying the events/objects in the current audiovisual input into different classes similar to AVHBench [ 36 ] .

Figure 12: QA Generation Verification Prompt - Stage 2.3. used to verify the automatic QAs generated for video-driven audio hallucination and audio-driven video hallucination.

Figure 13: Preference Data Generation Prompt - Stage 3 (Captioning) used to generate preference data pairs for the captioning task for training MoD-DPO.

Figure 14: Preference Data Generation Prompt - Stage 3 (Presence) used to generate preference data pairs for the object/event presence task for training MoD-DPO.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,084 papers

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Jan 29, 2026Sangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs
Oct 19, 2026Zijian Liu, Chaoyou Fu, Yan Yang +2LLM Hallucination Mitigation Multimodal Hallucination
School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China. · University of Chinese Academy of Sciences, Beijing 100049, China.
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
Sep 10, 2026Jianjiang Yang, Peihang Li, Shanqing Xu +3Hallucination Detection in VLMs Hallucination Detection
The University of Manchester · The University of Hong Kong · Huazhong University of Science and Technology +2

Jan 29, 2026J/K move · Enter open · S save

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro

Integrated Vision Language Lab, KAIST, South Korea

Multimodal Robustness Hallucination in Language Models Multimodal Large Language Models Multimodal Hallucination Contrastive Decoding Multimodal Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD

Abstract

Multimodal Large Language Models

Explore similar work

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs

OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models