Omnimodal large language models (Omni-LLMs) integrate audio, video, and text for multimodal reasoning, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoding methods mitigate these errors through modality perturbation or relevance-based weighting. However, due to interference from irrelevant modality cues during fusion and difficulty detecting shared errors, these methods may suppress correct unimodal predictions and allow erroneous consensus to persist. To address these limitations, we propose Provenance-Aware Consensus Decoding (PACD), a training-free framework for mitigating cross-modal hallucinations. PACD comprises two complementary paths based on whether the branches reach consensus. For non-consensus cases, Relevance-Guided Candidate Arbitration identifies task-relevant unimodal evidence and selectively arbitrates between predictions using alternative candidates. For consensus cases, Provenance-Based Consensus Recovery leverages the outranking principle to evaluate the consistency and strength of layer-wise modality evidence for alternative answers, selectively resolving erroneous consensus under confidence constraints. Extensive experiments on four Omni-LLMs and five benchmarks across two tasks demonstrate the superiority of PACD. Compared to existing methods, PACD achieves up to 27.6% relative improvement over standard decoding across CMM, AVHBench, and SVHalluc for cross-modal hallucination mitigation, and an average relative improvement of 7.7% over best baseline across OmniVideoBench and FutureOmni for general audio-visual reasoning.
Figures & tables
Figure 1: PACD achieves the state-of-the-art on CMM, AVHBench, and SVHalluc across Qwen2.5-Omni-7B and OmniVinci.
Figure 2: Motivation of PACD. (a) Correct modality-specific predictions may be lost during fusion. (b) Incorrect consensus among modality-specific branches often remains unresolved by MAD.
Figure 3: Overview of PACD. RGCA preserves reliable modality-specific predictions through relevance-guided routing and provenance-based arbitration. PBCR leverages layer-wise evidence to selectively correct incorrect consensus.
Method
CMM
AVHBench
SVHalluc
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Sem.
Temp.
Overall Acc.
VideoLLaMA2-AV-7B
Base
72.3
82.3
77.3
70.8
76.1
72.6
57.2
43.8
51.7
AVCD
69.8
84.8
77.3
78.0
80.1
78.7
62.5
45.2
55.4
MAD
81.3
80.5
80.9
79.7
77.0
78.8
65.1
44.5
56.7
PACD
86.5
78.3
82.4 ( ↑ 6.6%)
81.2
79.4
80.6 ( ↑ 11.0%)
71.0
45.0
60.4 ( ↑ 16.8%)
Table 1: Main results on cross-modal hallucination benchmarks. We compare the base decoding strategy (Base), AVCD, MAD, and our proposed PACD. Results are reported in accuracy (%). Arrows indicate relative gains over Base.
Method
CMM
AVHBench
SVHalluc
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Sem.
Temp.
Overall Acc.
Base
81.0
83.8
82.4
72.2
68.8
71.1
54.5
45.8
50.9
Uniform
86.0
85.8
85.9
76.2
68.6
73.7
57.0
46.2
52.6
Argmax
86.3
82.3
84.3
74.7
68.8
72.7
59.6
45.1
53.6
MAD
86.0
84.8
85.4
77.6
70.2
75.1
59.2
46.3
53.9
PACD
88.5
86.3
87.4
82.6
71.0
78.8
75.7
46.4
63.7
Table 2: Comparison of decoding strategies on CMM, AVHBench, and SVHalluc using OmniVinci.
Method
CMM
AVHBench
SVHalluc
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Sem.
Temp.
Overall Acc.
Base
81.0
83.8
82.4
72.2
68.8
71.1
54.5
45.8
50.9
Routing
87.8
79.3
83.5
77.7
67.6
74.3
59.7
44.9
53.6
RGCA
88.3
85.5
86.9
81.6
69.8
77.7
59.8
45.5
54.0
R+PBCR
88.0
80.0
84.0
78.7
69.3
75.6
75.0
45.4
62.9
PACD
88.5
86.3
87.4
82.6
71.0
78.8
75.7
46.4
63.7
Table 3: Component ablation of PACD on CMM, AVHBench, and SVHalluc using OmniVinci.
Method
OmniVideoBench
FutureOmni
VideoLLaMA2-AV
27.5
38.8
VideoLLaMA2-AV+MAD
28.3
36.3
[][] VideoLLaMA2-AV+PACD
31.1 ( ↑ 9.9%)
40.0 ( ↑ 10.2%)
OmniVinci
32.9
52.7
OmniVinci+MAD
32.5
51.7
[][] OmniVinci+PACD
34.8 ( ↑ 7.1%)
53.5 ( ↑ 3.5%)
Table 4: Generalization performance on OmniVideoBench and FutureOmni. Arrows indicate relative gains over MAD.
CMM
AVHBench
Method
Latency
GPU Mem.
Latency
GPU Mem.
AVCD
1.70s
22.6G
1.40s
22.7G
MAD
1.41s
20.7G
1.40s
20.7G
PACD
1.45s
20.7G
1.44s
20.7G
Table 5: Computational efficiency on CMM and AVHBench using VideoLLaMA2-AV-7B.
Figure 4: Performance across video durations and incorrect-consensus correction on SVHalluc using OmniVinci.
Control
Meaning and implementation role
N
Number of decoder layers included in provenance evidence aggregation.
τT
Confidence threshold for determining the preference of the routed target branch in RGCA.
τM
Confidence threshold for determining the preference of the MAD prediction in RGCA.
τC
Minimum cross-layer provenance consistency required for PBCR recovery.
τS
Minimum provenance support strength required for PBCR recovery.
τR
Confidence threshold governing selective consensus recovery in PBCR.
Table 6: PACD hyperparameters and their implementation roles, with the reference configuration N=10 , τT=1.2 , τM=3.0 , τC=0.2 , τS=−0.6 , and τR=2.8 .
Figure 5: Sensitivity to the number of examined layers N on CMM, AVHBench, and SVHalluc using OmniVinci. Insets provide enlarged views of the accuracy variations on CMM and AVHBench.
Figure 6: Hyperparameter sensitivity of RGCA and PBCR on CMM, AVHBench, and SVHalluc using OmniVinci. The upper heatmaps show the joint effects of τT and τM , while the bottom plot shows the average accuracy under different τC values. Stars indicate the reference configurations.
Figure 7: Joint sensitivity of PBCR to the provenance-strength threshold τS and recovery-confidence threshold τR on CMM, AVHBench, and SVHalluc using OmniVinci. The star indicates the reference configuration τS=−0.6 and τR=2.8 .
Method
CMM
AVHBench
SVHalluc
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Sem.
Temp.
Overall Acc.
VideoLLaMA2-AV-7B
Base
72.3
82.3
77.3
70.8
76.1
72.6
57.2
43.8
51.7
Uniform
77.5
82.8
80.1
77.1
77.2
77.1
61.5
43.9
54.3
Argmax
80.0
71.3
75.6
77.4
75.9
76.9
64.1
44.5
56.0
MAD
81.3
80.5
80.9
79.7
77.0
78.8
65.1
44.5
56.7
Table 7: Complete comparison with simplified decoding strategies. Accuracy (%) on CMM, AVHBench, and SVHalluc for all four backbones. Uniform assigns equal modality relevance within the MAD combination rule; Argmax converts modality relevance into a one-hot assignment and applies the same MAD combination rule. Base denotes the original joint audio-visual generation output. Overall accuracy is question-weighted. Bold indicates the highest displayed result within each model and metric, including ties after rounding.
Method
Sem.
Temp.
Avg.
GSA
FGSA
CMSB
TA
TF
CMTB
VideoLLaMA2-AV-7B
Base
50.0
57.2
63.1
50.0
26.8
51.0
51.7
AVCD
50.0
67.1
69.1
50.0
29.7
52.5
55.4
MAD
50.0
69.6
74.0
50.0
29.7
50.4
56.7
PACD
50.0
82.4
79.0
50.0
32.6
49.6
60.4
Table 8: SVHalluc task-level comparison with AVCD and MAD. Accuracy (%) on all six tasks. GSA, FGSA, and CMSB denote Global Semantic Alignment, Fine-Grained Semantic Alignment, and Cross-Modal Semantic Binding; TA, TF, and CMTB denote Temporal Alignment, Temporal Forecasting, and Cross-Modal Temporal Binding. The first three tasks form the semantic category (Sem.) and the last three form the temporal category (Temp.). Avg. is question-weighted accuracy over all 2,405 questions. Bold denotes the highest displayed result within each backbone and column, including ties after rounding.
Method
Sem.
Temp.
Avg.
GSA
FGSA
CMSB
TA
TF
CMTB
VideoLLaMA2-AV-7B
Base
50.0
57.2
63.1
50.0
26.8
51.0
51.7
Uniform
50.0
65.1
68.0
50.0
29.3
49.3
54.3
Argmax
50.0
60.1
79.0
50.0
29.0
51.0
56.0
MAD
50.0
69.6
74.0
50.0
29.7
50.4
56.7
Table 9: Task-level comparison of simplified decoding strategies on SVHalluc. Uniform assigns equal modality relevance, while Argmax converts relevance into a one-hot assignment; both apply the MAD combination rule. Task abbreviations follow Table 8 ; Avg. is accuracy over all 2,405 questions, rather than the unweighted mean of the six displayed task accuracies. All values are percentages. Bold denotes the highest displayed result within each backbone and column, including ties after rounding.
Method
CMM
AVHBench
SVHalluc
Visual Dom.
Audio Dom.
Overall Acc.
Video-Driven Audio Hall.
Audio-Driven Video Hall.
Overall Acc.
Sem.
Temp.
Overall Acc.
VideoLLaMA2-AV-7B
Base
72.3
82.3
77.3
70.8
76.1
72.6
57.2
43.8
51.7
Routing
81.0
68.0
74.5
78.2
76.5
77.6
64.0
43.3
55.6
RGCA
84.5
78.8
81.6
81.1
79.0
80.4
65.5
43.9
56.7
R+PBCR
83.0
67.5
75.3
78.4
76.6
77.8
71.1
43.7
59.9
Table 10: Complete component comparison of PACD. Accuracy (%) on both categories and the overall benchmark for CMM, AVHBench, and SVHalluc. Routing adopts cR ; RGCA adds provenance-guided candidate construction and confidence-based arbitration to obtain cB ; R+PBCR combines routing with consensus recovery while omitting that arbitration step. PACD combines RGCA and PBCR with the recovery priority defined in the main paper. Bold denotes the highest displayed result per model and metric, including ties after rounding.
Method
Sem.
Temp.
Avg.
GSA
FGSA
CMSB
TA
TF
CMTB
VideoLLaMA2-AV-7B
Base
50.0
57.2
63.1
50.0
26.8
51.0
51.7
Routing
50.0
60.1
78.8
50.0
31.2
46.0
55.6
RGCA
50.0
70.0
74.7
50.0
30.1
48.7
56.7
R+PBCR
50.0
81.5
80.0
50.0
32.2
46.3
59.9
Table 11: Task-level component comparison on SVHalluc. Routing, RGCA, R+PBCR, and PACD follow the definitions in Sec. 11 . Task abbreviations follow Table 8 . Avg. is question-weighted accuracy over all 2,405 questions. All values are percentages. Bold denotes the highest displayed result within each backbone and column, including ties after rounding.
Figure 8: Accuracy across prepared-input duration groups. Rows correspond to CMM, AVHBench, and SVHalluc, and columns correspond to the four backbones. Blue, purple, and green bars show Base, MAD, and PACD, respectively. Annotations next to PACD bars give its accuracy change from Base in percentage points.
Figure 9: Candidate-score discrimination and PACD operating points. Gray and blue curves represent ROC threshold sweeps of candidate-logit margins from the joint audio-visual branch (RAW) and MAD, respectively. Red stars indicate the operating points of PACD. Columns correspond to four Omni-LLMs, and rows correspond to CMM, AVHBench, and SVHalluc. CMM and AVHBench use binary ROC with Yes as the positive class, while SVHalluc uses pooled candidate-level one-versus-rest micro-ROC over two- and three-choice questions.
Figure 10: Stability of PACD’s accuracy improvements under media-level resampling. Dots indicate the accuracy gains of PACD over Base in percentage points, while horizontal bars represent 95% percentile intervals obtained from 5,000 paired media-cluster bootstrap replicates. Media IDs are sampled with replacement, with identical samples used for both methods. The dashed vertical line denotes zero improvement. All intervals lie above zero across the 12 model–benchmark combinations.
Figure 11: Qualitative examples of PACD preserving correct modality-specific predictions during fusion.
Figure 12: Qualitative examples of PACD correcting incorrect consensus across audio-only, video-only, and audio-visual branches.
Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.
Yuchen Deng, Chang Sun, Hai-Tao Zheng +2
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory · School of Cyber Science and Engineering, Zhengzhou University
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.
Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani
Recent Omni LLMs (OLLMs) have demonstrated remarkable capabilities in joint audio-visual reasoning, yet cross-modal hallucinations remain a major challenge. Existing mitigation methods mainly operate at the output level, overlooking the entanglement of grounding evidence and learned priors within fused representations. To address this issue, we propose Omni-Woodpecker (OWP), a training-free framework that shifts hallucination mitigation from output-level manipulation to conflict-conditioned representation rebalancing. Specifically, OWP first identifies cross-modal conflicts from complementary modality-specific views. It then employs orthogonal decomposition to separate grounding evidence from competing priors within the hidden states. Finally, OWP adaptively rebalances fused representations by suppressing misleading priors and restoring ignored evidence before decoding. Extensive experiments demonstrate the effectiveness and strong cross-architecture generalization of OWP. Using Qwen2.5-Omni-7B as the backbone, OWP improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, while consistently achieving gains of 0.67-2.25 points across four general OLLM reasoning benchmarks. Furthermore, OWP generalizes effectively to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks. Code is publicly available at https://github.com/ZiiJianLiu/Omni-Woodpecker
Zijian Liu, Chaoyou Fu, Yan Yang +2
School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China. · University of Chinese Academy of Sciences, Beijing 100049, China.