Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs
Authors: Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He
Organizations: School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China. · University of Chinese Academy of Sciences, Beijing 100049, China.
Recent Omni LLMs (OLLMs) have demonstrated remarkable capabilities in joint audio-visual reasoning, yet cross-modal hallucinations remain a major challenge. Existing mitigation methods mainly operate at the output level, overlooking the entanglement of grounding evidence and learned priors within fused representations. To address this issue, we propose Omni-Woodpecker (OWP), a training-free framework that shifts hallucination mitigation from output-level manipulation to conflict-conditioned representation rebalancing. Specifically, OWP first identifies cross-modal conflicts from complementary modality-specific views. It then employs orthogonal decomposition to separate grounding evidence from competing priors within the hidden states. Finally, OWP adaptively rebalances fused representations by suppressing misleading priors and restoring ignored evidence before decoding. Extensive experiments demonstrate the effectiveness and strong cross-architecture generalization of OWP. Using Qwen2.5-Omni-7B as the backbone, OWP improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, while consistently achieving gains of 0.67-2.25 points across four general OLLM reasoning benchmarks. Furthermore, OWP generalizes effectively to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks. Code is publicly available at https://github.com/ZiiJianLiu/Omni-Woodpecker
Figures & tables
Fig. 1: Comparison of omni-modal hallucination mitigation paradigms. (a) Existing output-level methods correct hallucinations after multimodal fusion. They lack direct control over the evidence-prior interactions already encoded in the fused representations. (b) Omni-Woodpecker (OWP) diagnoses conflicts using complementary evidence views. It orthogonally separates grounding evidence from competing priors, and adaptively restores their balance within the fused representations before decoding.
Fig. 2: Overview of Omni-Woodpecker (OWP). Given a query and an audio-visual input, OWP runs four views of a frozen model to find the target modality and the competing cue. It then splits token–head responses into protected evidence, competing prior, and unchanged context. Finally, it divides one correction budget between path reweighting and residual-state restoration and writes the edited state back to the same layer. Valid context is kept, and the cue that causes the hallucination is suppressed.
Model / Method
CMM
AVHBench
Visual
Audio
Lang.
Overall
Audio → Video
Video → Audio
AV Match.
Overall
Acc.
Acc.
Acc.
Acc.
PA
HR
Acc.
Acc.
Acc.
Acc.
F1
External omni backbones
VITA-1.5
52.0
71.5
56.8
60.1 ( −13.1 )
78.3
41.8
68.6
57.9
47.9
56.7 ( −18.4 )
60.6
OmniVinci
81.8
82.3
78.5
80.8 ( +7.6 )
86.8
74.8
70.7
69.7
64.0
67.9 ( −7.2 )
68.6
Qwen3-Omni
83.3
79.3
82.8
81.8 ( +8.6 )
91.3
72.2
84.0
80.1
70.4
77.5 ( +2.4 )
79.7
TABLE I: Main results on CMM and AVHBench (%). PA/HR: CMM perception accuracy/hallucination resistance. F1: AVHBench official binary F1. All rows, including external backbones run from released checkpoints, follow the protocol of Section IV-A . Bold: best within each group. Parentheses: overall change against the group backbone (against Qwen2.5-Omni-7B for external models).
Model
OmniB.
WorldS.
MUSIC-AVQA
AVQA
Qwen2.5-Omni-7B
45.36
45.21
85.08
91.83
+ OWP
47.29 (+1.93pp)
47.46 (+2.25pp)
86.08 (+1.00pp)
92.50 (+0.67pp)
TABLE II: General omni-modal reasoning accuracy (%) of Qwen2.5-Omni-7B with and without OWP. Parentheses: gain over the backbone.
Method
Qwen2.5- Omni-7B
VideoLLaMA2-AV
Lat. (s)
Rel.
Calls
Lat. (s)
Rel.
Backbone
3.54
1.00 ×
1
1.40
1.00 ×
MAD
9.96
2.81 ×
9
3.29
2.36 ×
HulluEdit
13.02
3.68 ×
5
1.94
1.38 ×
OWP
6.05
1.71 ×
≤ 5
2.48
1.77 ×
TABLE III: Inference cost per input: latency, latency relative to the backbone, and forward calls. OWP uses the conflict-triggered schedule, timed per component.
Variant
Evi.
Pri.
Res.
Pth.
Ada.
AVH
PA
HR
Backbone
✗
✗
✗
✗
✗
75.51
89.50
57.00
Evidence
✓
✗
✓
✓
✓
80.92 ( +5.41 )
83.67 ( −5.83 )
70.00 ( +13.00 )
Prior
✗
✓
✓
✓
✓
82.23 ( +6.72 )
85.33 ( −4.17 )
84.33 ( +27.33 )
Residual
✓
✓
✓
✗
✗
83.23 ( +7.72 )
85.67 ( −3.83 )
84.33 ( +27.33 )
Path
✓
✓
✗
✓
✗
81.68 ( +6.17 )
84.67 ( −4.83 )
69.67 ( +12.67 )
Fixed
✓
✓
✓
✓
✗
83.93 ( +8.42 )
84.67 ( −4.83 )
82.33 ( +25.33 )
TABLE IV: Ablation of OWP components, intervention sites, and allocation strategies on Qwen2.5-Omni-7B (protocol of Table I ). ✓/✗: enabled/disabled. AVH: directional AVHBench accuracy. Bold: best per column.
Input
Qwen2.5- Omni-7B
VideoLLaMA2-AV
Full (backbone)
82.06
76.85
One physical stream, same for every row
Audio only
69.71 ( −12.35 )
69.91 ( −6.94 )
Video only
71.42 ( −10.64 )
66.15 ( −10.70 )
Text only
52.88 ( −29.18 )
53.61 ( −23.24 )
Stream chosen per row by the oracle target modality
TABLE V: Input-view ablation (shared executor, binary yes/no probe) on all CMM and directional AVHBench rows with a single target modality. The frozen backbone is re-run on the indicated stream with the unchanged prompt; the oracle block uses the benchmark-defined target modality. The shift row adds a constant to the yes-minus-no margin to match OWP’s yes rate. OWP edits the full input. Parentheses: paired difference against the full input (pp).
ID
Diagnostic instruction appended to the query
1
Identify which modality is required to answer the question (audio, video, or both).
2
Given this question, select the necessary modality for reasoning (audio, video, or both).
3
Which modality does this question require (audio, video, or both)?
4
State the modality relevant for answering this question (audio, video, or both).
TABLE VI: The four phrasings of the diagnostic query used for target typing. Each phrasing is appended to the original query and keeps the same three response tokens.
Fig. 3: Mechanism analysis of OWP. (a,b) Change in overall accuracy, AVH (directional AVHBench) accuracy, and CMM HR when evidence enhancement (a) or prior suppression (b) is applied at a single layer; bands show paired-bootstrap 95% CIs, and open circles mark the layers that OWP edits. (c) Evidence-minus-prior balance before (frozen backbone) and after each method; positive values denote evidence-dominant states. (d) Answer-change rate when the query-relevant (vertical) or irrelevant (horizontal) stream is corrupted (1,000 samples); hollow and filled markers denote mild and strong corruption, and points farther above the dashed y=x line respond more selectively. (e) AVHBench accuracy against CMM HR for OWP and direction controls on the conflict subset that OWP edits; fill encodes CMM PA, error bars show row-bootstrap 95% CIs, and small markers show three projection-matched random seeds. Only OWP, with both semantic directions intact, reaches the high-accuracy, high-HR region.
Fig. 4: Qualitative comparison on reciprocal visual- and audio-grounded queries; pink and green boxes mark incorrect and correct answers. Left: given audible crowing, MAD and HulluEdit report a visually absent rooster. Right: given a visual spraying scene, they deny the concurrent speech. OWP answers both from the queried modality while keeping compatible context.
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.
Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani
Omni-modal large language models (OmniLLMs) unify text, images, audio, and video, yet hallucinate when generation relies on the wrong evidence. Existing inference-time methods can reduce hallucinations, but rarely reveal which evidence sustains a generated commitment. We introduce OmniConfess, a training-free method for mitigating omni-modal hallucinations. It fixes a candidate response and re-scores it at token resolution under controlled channel-wise evidence interventions, producing a structured token-by-channel confession that reveals the response's evidential dependence. OmniConfess uses this confession to preserve grounded content and correct commitments driven by irrelevant or contradictory evidence. To evaluate OmniConfess, we construct OmniHalluBench, a 3,540-example benchmark built from six datasets spanning text, image, audio, and video settings and both judgment and free-form generation. Experiments show that OmniConfess mitigates hallucinations across heterogeneous modality and task settings. Our code and benchmark are publicly available at https://github.com/RongHuiQiang/OmniConfess.
Huiqiang Rong, Haoran Luo, Hui Feng +4
Beijing University of Posts and Telecommunications · Nanyang Technological University, Singapore
Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as modality misrouting. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.
Yu Mi, Licheng Pan, Jiayi Zhu +7
Zhejiang University · Ant Group · Institute of Computing Technology, CAS