Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

Yu Mi; Licheng Pan; Jiayi Zhu; Qirui Liu; Yitong Qiao; Changtong Xu; Tao Wei; Fei Sun

cs.AIOct 13, 2026

Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

Authors: Yu Mi, Licheng Pan, Jiayi Zhu, Qirui Liu, Yitong Qiao, Changtong Xu, Tao Wei, Fei Sun,

Organizations: Zhejiang University · Ant Group · Institute of Computing Technology, CAS

Abstract

Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as modality misrouting. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.

Figures & tables

Figure 1: Overview of CaRe. Given audio-visual inputs and a question, CaRe uses a text-only modality probe to select a candidate modality, detects decision conflicts between the joint and counterfactual branches, and applies confidence-adaptive residual intervention at selected decoder layers with strengths decayed across depth.

Figure 2: Bidirectional modality misrouting under audio-visual interference. CaRe mitigates these errors by steering the fused representation toward the selected modality-specific branch.

Figure 3: Target-branch recovery among incorrect joint predictions on CMM and AVHBench (%).

		CMM					AVHBench
Backbone	Method	Visual Dominance		Audio Dominance		Overall Acc.	Video-Driven Audio Hall.		Audio-Driven Video Hall.		Overall Acc.
		Acc.	F1	Acc.	F1	Overall Acc.	Acc.	F1	Acc.	F1	Overall Acc.
Qwen2.5- Omni-7B	Baseline	69.3	75.7	71.8	77.0	70.5	67.6	75.3	74.1	77.4	69.8
	+VCD	75.5	79.4	66.5	73.5	71.0	68.1	75.4	77.4	78.0	71.2
	+ICT	75.3	79.1	66.5	73.6	70.9	68.8	75.6	76.7	78.4	71.4
	+AVCD	67.8	74.9	70.5	76.2	69.1	77.9	81.4	81.7	82.8	79.2

Table 1: Main results on CMM and AVHBench (%). We evaluate Qwen2.5-Omni-7B and MiniCPM-o-2.6 on the Visual Dominance and Audio Dominance subsets of CMM, and on the Video-driven Audio Hallucination and Audio-driven Video Hallucination tasks of AVHBench. We report Accuracy (Acc.) and F1, with Yes treated as the positive class. Overall Acc. is computed over all samples in the two reported subsets or tasks. Best results are highlighted in bold.

Backbone	Method	CMM	AVHBench
Backbone	Method	Acc.	Acc.
Qwen2.5- Omni-3B	Base	76.9	76.4
Qwen2.5- Omni-3B	+CaRe	82.5 $(+5.6)$	81.0 $(+4.6)$
Qwen2.5- Omni-7B	Base	70.5	69.8
Qwen2.5- Omni-7B	+CaRe	85.9 $(+15.4)$	82.3 $(+12.5)$

Table 2: Scaling robustness on Qwen2.5-Omni models. Gains over Base are shown in parentheses.

Backbone	Method	CMM		AVHBench
Backbone	Method	Visual Dom.	Audio Dom.	V $\to$ A Hall.	A $\to$ V Hall.
Qwen2.5-Omni-7B	CF-SELECT	80.14	86.46	54.22	53.61
Qwen2.5-Omni-7B	+CaRe	81.69 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+1.55)}}$	88.54 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+2.08)}}$	63.05 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+8.84)}}$	62.89 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+9.28)}}$
MiniCPM-o-2.6	CF-SELECT	66.67	84.43	48.56	60.38
MiniCPM-o-2.6	+CaRe	73.02 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+6.35)}}$	86.49 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+2.06)}}$	59.38 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+10.82)}}$	64.15 ${}_{\,{\color[rgb]{0.2344,0.4297,0.668}(+3.77)}}$

Table 3: Residual intervention vs. direct branch selection on gate-positive samples (%). Evaluations are conducted exclusively on conflict-gated samples (

g=1

). CF-SELECT directly outputs the counterfactual unimodal prediction

\hat{y}^{r}

, whereas CaRe applies continuous residual steering to the joint representation. Accuracy gains of CaRe over CF-SELECT are reported in parentheses at the bottom right.

Figure 7: Robustness of text-only modality routing. Probe agreement (%) with the task-defined target modality across prompt formulations and all six modality-label orderings. Results are shown for both backbones on CMM and AVHBench; prompt templates are provided in Appendix D .

Backbone	Method	All	Question Category						Duration
Backbone	Method	All	Event Seq.	AV Event Align.	Inference	Reasoning	Context Understanding	Comparative	30s	60s
Qwen2.5-Omni-7B	Base	61.5	55.6	50.4	75.3	75.4	57.5	66.4	62.6	60.2
Qwen2.5-Omni-7B	CaRe	61.8	56.2	51.3	76.6	76.6	56.5	64.9	62.9	60.5
MiniCPM-o-2.6	Base	61.0	55.6	48.7	76.0	72.6	60.6	63.4	61.1	60.9
MiniCPM-o-2.6	CaRe	61.5	55.9	48.7	77.3	73.7	60.6	64.1	61.8	61.1

Table 4: Accuracy on Daily-Omni (%). Results are reported across question categories and video durations. The better result within each backbone is highlighted in bold.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Appendix	Title
Appendix A	Additional Ablation and Hyperparameter Analysis
$\hookrightarrow$ Appendix A.1	Ablation of the Counterfactual Conflict Gate
$\hookrightarrow$ Appendix A.2	Impact of the Intervention Strength Scale $\lambda_{\alpha}$
$\hookrightarrow$ Appendix A.3	Impact of the Confidence Threshold $\eta$
$\hookrightarrow$ Appendix A.4	Joint Sensitivity to $\lambda_{\alpha}$ and $\eta$
Appendix B	Scaling Analysis

Appendix

Table 10

Backbone	Intervention Coverage		Accuracy	Extra Interventions (Help / Harm)
Backbone	Conflict-gated	Routing-only	Conflict / Routing	Extra Interventions (Help / Harm)
Qwen2.5-Omni-7B	167/800 (20.9%)	773/800 (96.6%)	85.9 / 85.9	606 (0 / 0)
MiniCPM-o-2.6	174/800 (21.8%)	772/800 (96.5%)	86.9 / 86.9	598 (0 / 0)

Appendix

Table 5: Ablation of the counterfactual conflict gate on CMM. Routing-only applies residual correction whenever the probe returns a unique unimodal candidate. Help/Harm counts additional routing-only interventions that change an originally incorrect/correct prediction into a correct/incorrect one, respectively.

Figure 8: Sensitivity to the intervention strength scale

\lambda_{\alpha}

on CMM. Lines report accuracy, while bars show the numbers of helpful and harmful prediction changes. Stars denote the default setting

\lambda_{\alpha}=1

Figure 9: Sensitivity to the confidence threshold

\eta

on CMM. Lines report accuracy, while bars show the numbers of samples assigned to the weak and strong intervention regimes. Stars denote the default settings,

\eta=0.2

for Qwen2.5-Omni-7B and

\eta=0.3

for MiniCPM-o-2.6.

Figure 10: Joint sensitivity to the intervention strength scale $\lambda_{\alpha}$ and confidence threshold $\eta$ on CMM. Each cell reports accuracy (%); darker colors indicate higher accuracy.

		CMM					AVHBench
Backbone	Method	Visual Dominance		Audio Dominance		Overall Acc.	Video-Driven Audio Hall.		Audio-Driven Video Hall.		Overall Acc.
		Acc.	F1	Acc.	F1	Overall Acc.	Acc.	F1	Acc.	F1	Overall Acc.
Qwen2.5-Omni-3B	Base	69.3	66.5	84.5	82.5	76.9	74.2	74.5	80.8	77.5	76.4
Qwen2.5-Omni-3B	+CaRe	79.8	79.1	85.3	83.5	82.5	81.0	81.4	80.9	77.6	81.0

Appendix

Table 6: Detailed scaling results on Qwen2.5-Omni-3B (%). We compare the base model with CaRe on CMM and AVHBench. Results are reported as Accuracy (Acc.) and F1, with Yes treated as the positive class. Overall Acc. is computed over all samples in the corresponding subsets or tasks. Best results are highlighted in bold.

Backbone	Dataset	Intervention Layers	$\gamma$	$\eta$	$\alpha^{W}$	$\alpha^{S}$
Qwen2.5-Omni-7B	CMM	25–27	0.5	0.2	$[0.2,0.5]$	$[0.5,0.8]$
Qwen2.5-Omni-7B	AVHBench	25–27	0.5	0.2	$[0.2,0.5]$	$[0.5,0.8]$
Qwen2.5-Omni-3B	CMM	33–35	0.5	0.2	$[0.2,0.5]$	$[0.5,0.8]$
Qwen2.5-Omni-3B	AVHBench	33–35	0.5	0.2	$[0.2,0.5]$	$[0.5,0.8]$
MiniCPM-o-2.6	CMM	25–27	0.5	0.3	$[0.1,0.3]$	$[0.5,0.8]$
MiniCPM-o-2.6	AVHBench	25–27	0.5	0.5	$[0.1,0.3]$	$[0.3,0.5]$

Appendix

Table 7: Hyperparameter configurations of CaRe on CMM and AVHBench.

ID	Prompt template
D	To answer this question, which modality is needed ({options}):
N1	Determine the minimum evidence needed to answer the question. Select one modality: {options}:
N2	What kind of input is necessary for a correct answer? Choose from: {options}:
N3	Classify the question by the information source it requires. Answer with one of: {options}:

Appendix

Table 8: Prompt formulations for the text-only modality probe. D is used throughout the main experiments, while N1–N3 are alternative formulations considered in the robustness analysis.

Backbone	Method	Mean	Std.	Median	P95
Qwen2.5-Omni-7B	CaRe	6.698	4.965	5.438	15.934
	MAD	16.612	25.150	8.797	50.130
	AVCD	4.838	2.852	4.262	11.022
MiniCPM-o-2.6	CaRe	4.571	4.053	3.416	12.075
	MAD	7.816	4.157	6.245	16.573
	AVCD	2.966	1.973	2.142	7.636

Appendix

Table 9: End-to-end inference latency under serial execution. Latency is measured over 200 samples for each backbone. We report the mean, standard deviation, median, and 95th-percentile (P95) latency in seconds.

Backbone	Probe	Joint AV	Counterfactual	Gate/Control	Residual Corr.
Qwen2.5-Omni-7B	0.021	4.369	2.004	$<0.001$	0.305
MiniCPM-o-2.6	0.021	3.031	1.479	$<0.001$	0.040

Appendix

Table 10: Average per-sample latency breakdown of CaRe. All values are reported in seconds and averaged over the same 200-example evaluation subset. Counterfactual denotes inference with the selected modality-specific branch, and Residual Corr. denotes the live residual-intervention stage.

Backbone	Method	Serial	Parallel	Reduction
Qwen2.5-Omni-7B	CaRe	6.698	4.782	28.61%
	MAD	16.612	12.038	27.53%
	AVCD	4.838	4.268	11.80%
MiniCPM-o-2.6	CaRe	4.571	3.239	29.15%
	MAD	7.816	5.374	31.25%
	AVCD	2.966	2.910	1.89%

Appendix

Table 11: Theoretical latency under ideal branch-level parallel execution. Serial latency is measured directly, while parallel latency is estimated from the measured stage-wise runtimes assuming independent branches can execute concurrently without scheduling, communication, or device-synchronization overhead. Reduction is relative to serial execution.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,084 papers

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models
Aug 26, 2026Yuchen Deng, Chang Sun, Hai-Tao Zheng +2Multimodal Large Language Models Multimodal Hallucination
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University
Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
Sep 29, 2026Yueran Ma, Ronghao LinAudio QA Multimodal Large Language Models
The University of Queensland, Brisbane, Australia · Shenzhen University, Shenzhen, China
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
Mar 3, 2026Ashutosh Chaubey, Jiacheng Pang, Mohammad SoleymaniMultimodal Large Language Models Multimodal Hallucination
University of Southern California

Aug 26, 2026J/K move · Enter open · S save

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Yuchen Deng, Chang Sun, Hai-Tao Zheng, Feidiao Yang+1

Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University

Multimodal Large Language Models Multimodal Hallucination

Open paper AskPDF arXiv

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.

Abstract

Multimodal Large Language Models

Explore similar work

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization