Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as modality misrouting. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.
Figures & tables
Figure 1: Overview of CaRe. Given audio-visual inputs and a question, CaRe uses a text-only modality probe to select a candidate modality, detects decision conflicts between the joint and counterfactual branches, and applies confidence-adaptive residual intervention at selected decoder layers with strengths decayed across depth.
Figure 2: Bidirectional modality misrouting under audio-visual interference. CaRe mitigates these errors by steering the fused representation toward the selected modality-specific branch.
Figure 3: Target-branch recovery among incorrect joint predictions on CMM and AVHBench (%).
CMM
AVHBench
Backbone
Method
Visual Dominance
Audio Dominance
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Qwen2.5- Omni-7B
Baseline
69.3
75.7
71.8
77.0
70.5
67.6
75.3
74.1
77.4
69.8
+VCD
75.5
79.4
66.5
73.5
71.0
68.1
75.4
77.4
78.0
71.2
+ICT
75.3
79.1
66.5
73.6
70.9
68.8
75.6
76.7
78.4
71.4
+AVCD
67.8
74.9
70.5
76.2
69.1
77.9
81.4
81.7
82.8
79.2
Table 1: Main results on CMM and AVHBench (%). We evaluate Qwen2.5-Omni-7B and MiniCPM-o-2.6 on the Visual Dominance and Audio Dominance subsets of CMM, and on the Video-driven Audio Hallucination and Audio-driven Video Hallucination tasks of AVHBench. We report Accuracy (Acc.) and F1, with Yes treated as the positive class. Overall Acc. is computed over all samples in the two reported subsets or tasks. Best results are highlighted in bold.
Backbone
Method
CMM
AVHBench
Acc.
Acc.
Qwen2.5- Omni-3B
Base
76.9
76.4
+CaRe
82.5 (+5.6)
81.0 (+4.6)
Qwen2.5- Omni-7B
Base
70.5
69.8
+CaRe
85.9 (+15.4)
82.3 (+12.5)
Table 2: Scaling robustness on Qwen2.5-Omni models. Gains over Base are shown in parentheses.
Table 3: Residual intervention vs. direct branch selection on gate-positive samples (%). Evaluations are conducted exclusively on conflict-gated samples ( g=1 ). CF-SELECT directly outputs the counterfactual unimodal prediction y^r , whereas CaRe applies continuous residual steering to the joint representation. Accuracy gains of CaRe over CF-SELECT are reported in parentheses at the bottom right.
Figure 7: Robustness of text-only modality routing. Probe agreement (%) with the task-defined target modality across prompt formulations and all six modality-label orderings. Results are shown for both backbones on CMM and AVHBench; prompt templates are provided in Appendix D .
Backbone
Method
All
Question Category
Duration
Event Seq.
AV Event Align.
Inference
Reasoning
Context Understanding
Comparative
30s
60s
Qwen2.5-Omni-7B
Base
61.5
55.6
50.4
75.3
75.4
57.5
66.4
62.6
60.2
CaRe
61.8
56.2
51.3
76.6
76.6
56.5
64.9
62.9
60.5
MiniCPM-o-2.6
Base
61.0
55.6
48.7
76.0
72.6
60.6
63.4
61.1
60.9
CaRe
61.5
55.9
48.7
77.3
73.7
60.6
64.1
61.8
61.1
Table 4: Accuracy on Daily-Omni (%). Results are reported across question categories and video durations. The better result within each backbone is highlighted in bold.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Appendix
Title
Appendix A
Additional Ablation and Hyperparameter Analysis
↪ Appendix A.1
Ablation of the Counterfactual Conflict Gate
↪ Appendix A.2
Impact of the Intervention Strength Scale λα
↪ Appendix A.3
Impact of the Confidence Threshold η
↪ Appendix A.4
Joint Sensitivity to λα and η
Appendix B
Scaling Analysis
Appendix
Table 10
Backbone
Intervention Coverage
Accuracy
Extra Interventions (Help / Harm)
Conflict-gated
Routing-only
Conflict / Routing
Qwen2.5-Omni-7B
167/800 (20.9%)
773/800 (96.6%)
85.9 / 85.9
606 (0 / 0)
MiniCPM-o-2.6
174/800 (21.8%)
772/800 (96.5%)
86.9 / 86.9
598 (0 / 0)
Appendix
Table 5: Ablation of the counterfactual conflict gate on CMM. Routing-only applies residual correction whenever the probe returns a unique unimodal candidate. Help/Harm counts additional routing-only interventions that change an originally incorrect/correct prediction into a correct/incorrect one, respectively.
Figure 8: Sensitivity to the intervention strength scale λα on CMM. Lines report accuracy, while bars show the numbers of helpful and harmful prediction changes. Stars denote the default setting λα=1 .
Figure 9: Sensitivity to the confidence threshold η on CMM. Lines report accuracy, while bars show the numbers of samples assigned to the weak and strong intervention regimes. Stars denote the default settings, η=0.2 for Qwen2.5-Omni-7B and η=0.3 for MiniCPM-o-2.6.
Figure 10: Joint sensitivity to the intervention strength scale λα and confidence threshold η on CMM. Each cell reports accuracy (%); darker colors indicate higher accuracy.
CMM
AVHBench
Backbone
Method
Visual Dominance
Audio Dominance
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Qwen2.5-Omni-3B
Base
69.3
66.5
84.5
82.5
76.9
74.2
74.5
80.8
77.5
76.4
+CaRe
79.8
79.1
85.3
83.5
82.5
81.0
81.4
80.9
77.6
81.0
Appendix
Table 6: Detailed scaling results on Qwen2.5-Omni-3B (%). We compare the base model with CaRe on CMM and AVHBench. Results are reported as Accuracy (Acc.) and F1, with Yes treated as the positive class. Overall Acc. is computed over all samples in the corresponding subsets or tasks. Best results are highlighted in bold.
Backbone
Dataset
Intervention Layers
γ
η
αW
αS
Qwen2.5-Omni-7B
CMM
25–27
0.5
0.2
[0.2,0.5]
[0.5,0.8]
AVHBench
25–27
0.5
0.2
[0.2,0.5]
[0.5,0.8]
Qwen2.5-Omni-3B
CMM
33–35
0.5
0.2
[0.2,0.5]
[0.5,0.8]
AVHBench
33–35
0.5
0.2
[0.2,0.5]
[0.5,0.8]
MiniCPM-o-2.6
CMM
25–27
0.5
0.3
[0.1,0.3]
[0.5,0.8]
AVHBench
25–27
0.5
0.5
[0.1,0.3]
[0.3,0.5]
Appendix
Table 7: Hyperparameter configurations of CaRe on CMM and AVHBench.
ID
Prompt template
D
To answer this question, which modality is needed ({options}):
N1
Determine the minimum evidence needed to answer the question. Select one modality: {options}:
N2
What kind of input is necessary for a correct answer? Choose from: {options}:
N3
Classify the question by the information source it requires. Answer with one of: {options}:
Appendix
Table 8: Prompt formulations for the text-only modality probe. D is used throughout the main experiments, while N1–N3 are alternative formulations considered in the robustness analysis.
Backbone
Method
Mean
Std.
Median
P95
Qwen2.5-Omni-7B
CaRe
6.698
4.965
5.438
15.934
MAD
16.612
25.150
8.797
50.130
AVCD
4.838
2.852
4.262
11.022
MiniCPM-o-2.6
CaRe
4.571
4.053
3.416
12.075
MAD
7.816
4.157
6.245
16.573
AVCD
2.966
1.973
2.142
7.636
Appendix
Table 9: End-to-end inference latency under serial execution. Latency is measured over 200 samples for each backbone. We report the mean, standard deviation, median, and 95th-percentile (P95) latency in seconds.
Backbone
Probe
Joint AV
Counterfactual
Gate/Control
Residual Corr.
Qwen2.5-Omni-7B
0.021
4.369
2.004
<0.001
0.305
MiniCPM-o-2.6
0.021
3.031
1.479
<0.001
0.040
Appendix
Table 10: Average per-sample latency breakdown of CaRe. All values are reported in seconds and averaged over the same 200-example evaluation subset. Counterfactual denotes inference with the selected modality-specific branch, and Residual Corr. denotes the live residual-intervention stage.
Backbone
Method
Serial
Parallel
Reduction
Qwen2.5-Omni-7B
CaRe
6.698
4.782
28.61%
MAD
16.612
12.038
27.53%
AVCD
4.838
4.268
11.80%
MiniCPM-o-2.6
CaRe
4.571
3.239
29.15%
MAD
7.816
5.374
31.25%
AVCD
2.966
2.910
1.89%
Appendix
Table 11: Theoretical latency under ideal branch-level parallel execution. Serial latency is measured directly, while parallel latency is estimated from the measured stage-wise runtimes assuming independent branches can execute concurrently without scheduling, communication, or device-synchronization overhead. Reduction is relative to serial execution.
Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.
Yuchen Deng, Chang Sun, Hai-Tao Zheng +2
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University
Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.
Yueran Ma, Ronghao Lin
The University of Queensland, Brisbane, Australia · Shenzhen University, Shenzhen, China
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.
Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani