Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.
Figures & tables
Figure 1: Motivation of this study. (a) A correct prediction does not necessarily indicate successful hallucination mitigation. (b) When question-required evidence changes, the prediction should change accordingly. (c) When non-required evidence changes, it should remain unchanged. The base model violates both behaviors, whereas RED follows the evidence required by the question.
Figure 2: Overview of RED for an audio-required case. (i) Given a query, RED first identifies the evidence required to answer it, selecting among none, audio, video, audio + video, or audio-visual interaction. (ii) The frozen AV-LLM produces the audio-visual prediction and only the auxiliary predictions required by the selected evidence type, from which RED measures audio, video, and non-additive audio-visual contributions relative to the textual reference. (iii) The selected contribution augments the original audio-visual prediction to produce the final evidence-guided prediction.
Figure 3: Evidence following under modality interventions. Higher sensitivity and lower invariance violation indicate better evidence following. The dashed reference uses only the modality required by the question; its invariance violation is zero by construction because the intervened non-required modality is not used. Base receives both audio and video. Results are averaged over question-required evidence types and three backbones.
Method
CMM
AVHBench
SVHalluc
Vis.
Aud.
Lang.
Avg.
ADV
VDA
AVM
Avg.
Sem.
Temp.
Avg.
VideoLLaMA2.1-7B-AV
Base
73.5
84.0
72.5
82.8
80.4
76.5
57.8
71.6
63.0
43.5
53.3
VCD ( Leng et al., 2024 )
70.8
84.5
73.8
81.5
80.7
74.6
56.3
70.6
64.6
43.6
54.1
ICD ( Wang et al., 2024 )
73.5
84.2
72.5
82.6
80.7
77.0
57.8
71.8
63.2
43.4
53.3
CODE ( Kim et al., 2024 )
72.5
84.0
72.0
82.4
80.5
76.0
57.8
71.4
63.2
43.1
53.2
Table 1: Performance on audio–visual hallucination benchmarks. Accuracy (%) on CMM, AVHBench, and SVHalluc. Vis., Aud., and Lang. denote the CMM visual-, audio-, and language-dominance subsets. For SVHalluc, Sem. and Temp. average the semantic (GSA, FGSA, CMSB) and temporal (TA, TF, CMTB) tasks. Avg. is the unweighted mean over all tasks in each benchmark.
Figure 5
Q-guided
PMI
CMM
AVH
Acc.
AUC
Acc.
AUC
✗
✗
84.8
92.3
75.2
82.0
✓
✗
86.8
93.6
76.2
82.7
✗
✓
84.2
91.9
75.8
82.8
✓
✓
87.0
94.0
78.6
86.1
Table 3: Component analysis of evidence correction. We ablate question-guided evidence selection and PMI-based contribution decomposition with the remaining RED procedure fixed. Results are averaged over three backbones. AUC denotes AUROC.
Figure 7Figure 8
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A: Question-guided evidence selection. The frozen backbone identifies the required evidence using the question text alone.
Method
Centering
Acc.
AUROC
VideoLLaMA2.1-7B-AV
RED (Ours)
×
85.5
92.5
RED (Ours)
✓
85.6
92.8
Qwen2.5-Omni-3B
RED (Ours)
×
86.1
93.6
RED (Ours)
✓
86.2
93.6
Appendix
Table A: Effect of score centering on CMM. Task-macro accuracy and AUROC (%) for RED with and without centering.
Figure B: Sensitivity to centering sample size on CMM. We vary the number of unlabeled centering questions under a fixed sampling configuration. Dashed lines denote RED without centering; the dotted vertical line marks the default budget of 100 questions.
Figure C: Robustness to centering sample selection on CMM. Small circles show results from five random draws of 100 unlabeled questions. Diamonds and error bars indicate the mean and one sample standard deviation. Stars show the reported RED results, and squares denote RED without centering.
Figure D: Effect of choosing different evidence contributions. We keep the evidence selection and centering fixed and vary only the contribution added to the audio-visual score. The left panel includes all eligible groups, while the right panel focuses on Audio and Video groups and additionally compares adding the other unimodal contribution. No correction denotes the centered audio-visual score without an additional evidence contribution. Higher TPR and TNR are better.
Task
Evidence
+ΔA
+ΔV
+ΔI
Acc.
AUROC
Acc.
AUROC
Acc.
AUROC
VDA
A
81.27
89.24
70.39
76.89
46.38
44.60
ADV
V
77.99
84.26
83.63
88.82
64.08
68.20
AVM
A↔V
69.56
77.65
62.42
68.51
70.68
78.54
Appendix
Table B: Modality-specific and residual contributions on AVHBench. Accuracy and AUROC (%) with Qwen2.5-Omni-3B when adding the audio ( ΔA ), video ( ΔV ), or residual audio-visual ( ΔI ) contribution to the same centered audio-visual score. VDA, ADV, and AVM correspond to Audio ( A ), Video ( V ), and Relation ( A↔V ) evidence requirements, respectively. The centering offsets are fixed across variants. Bold indicates the best result in each row for each metric.
A+V subset
n
+ΔA
+ΔV
+(ΔA+ΔV)
+ΔI
Prompt-selected
1,786
36.73
38.63
38.19
33.82
Oracle-tagged
2,124
37.95
38.79
40.16
30.23
Appendix
Table C: Additive contributions for both-modality evidence on MUSIC-AVQA-Hard. Accuracy (%) with Qwen2.5-Omni-7B. Prompt-selected contains questions assigned to A+V by RED ’s question-guided evidence selection, whereas Oracle-tagged contains questions identified by the dataset’s Audio-Visual evidence tags. All variants use the same centering offsets and differ only in the contribution added to the centered audio-visual score. Bold indicates the best result in each row.
Figure E: Question-guided evidence selection across five benchmarks. For each backbone, proportions are computed over all questions in each evaluation split and then averaged equally across three backbones. Centers report the number of questions per backbone. Audio, Video, Both, Relation, and None select ΔA , ΔV , ΔA+ΔV , ΔI , and no additional correction, respectively. Both selects additive audio and video contributions; Relation selects their residual interaction. The distributions reflect question-only selector predictions, not oracle annotations. Nonzero categories below 1% are listed beneath each chart. MUSIC-AVQA uses the official corrected test annotations (9,129 questions); MUSIC-AVQA-Hard contains 2,508 questions from this split.
Stage
Time (ms)
Evidence selection
89.1
Media preparation
2372.1
Perceptual encoding
1881.4
Answer-prompt prefill
1381.7
Answer-token prediction
28.4
Score composition and overhead
37.2
Appendix
Table D: Runtime breakdown of RED . Stage times are averaged equally over CMM, AVHBench, and SVHalluc across three backbones using separate profiling runs.
Method
VideoLLaMA2.1-7B-AV
Qwen2.5-Omni-3B
Qwen2.5-Omni-7B
CMM
AVH
SVH
CMM
AVH
SVH
CMM
AVH
SVH
Base
1.0×
1.0×
1.0×
1.0×
1.0×
1.0×
1.0×
1.0×
1.0×
VCD ( Leng et al., 2024 )
2.2×
3.0×
1.6×
2.9×
3.2×
2.9×
3.0×
3.3×
3.0×
ICD ( Wang et al., 2024 )
2.1×
2.8×
1.6×
3.8×
4.1×
3.0×
3.7×
4.1×
3.2×
CODE ( Kim et al., 2024 )
2.7×
3.8×
2.0×
3.6×
3.8×
2.1×
4.0×
3.8×
2.2×
SID ( Huo et al., 2025 )
1.2×
1.4×
1.1×
1.5×
1.5×
1.5×
1.4×
1.4×
1.5×
Appendix
Table E: Relative TTFT across audio-visual hallucination benchmarks. Time to first token normalized to the corresponding Base ( × Base; lower is better).
Method
VL
AL
VAL
Vis.
Aud.
Lang.
Avg.
TTFT (s)
Relative
Base model
90.50
84.75
89.50
69.25
84.50
75.25
82.29
2.717
1.000×
SID (0% removed)
90.50
84.75
89.50
69.25
84.50
75.25
82.29
4.379
1.612×
SID (60% removed)
87.50
83.75
88.25
71.25
81.25
77.75
81.63
4.179
1.538×
SID (90% removed)
89.25
84.25
91.00
71.25
74.50
79.00
81.54
4.101
1.509×
RED (Ours)
88.25
84.75
95.00
76.25
88.50
77.25
85.00
3.893
1.433×
Appendix
Table F: Accuracy–TTFT trade-off of SID ( Huo et al., 2025 ) pruning on CMM. Results with Qwen2.5-Omni-3B under a common 2-fps configuration. Accuracy (%) is reported on all 2,400 questions, and TTFT is normalized to the paired Base. Bold indicates the best accuracy or lowest non-base TTFT.
Test
Req.
Swap
Gold
Donor
Pairs
Samples
Sens.
Audio
A
Yes → No
Annotated
804
804
No → Yes
Annotated
524
524
Video
V
Yes → No
Annotated
192
192
No → Yes
Annotated
101
101
Inv.
Video
A
unchanged
Random
674
674
Audio
V
unchanged
Random
1,064
1,064
Appendix
Table G: Construction of evidence interventions. Required-modality swaps change the answer, whereas irrelevant-modality swaps preserve it. Pairs denote retained interventions and Samples denote unique source video–question items; the total is deduplicated.
Figure F: Illustration of pair-level evidence intervention. For a video-required question, we replace the video of AVHBench sample 01161 with that of sample 00640 while retaining the original audio from 01161. The reference answer changes accordingly, and the pair succeeds only when both the original and intervened inputs are answered correctly. This is an illustrative pair, not a retained evaluation pair; waveforms are schematic.
Figure G: Evidence following under audio-visual interventions on AVHBench. Sensitivity requires both answers in an answer-changing pair to be correct, while invariance violation measures prediction changes under non-required-modality swaps. Bars and whiskers denote the mean and min–max range across three backbones.
Method
VL
AL
VAL
Vis.
Aud.
Lang.
Avg.
VideoLLaMA2.1-7B-AV
Base model
93.5
81.2
92.2
73.5
84.0
72.5
82.8
VCD ( Leng et al., 2024 )
92.5
77.5
89.8
70.8
84.5
73.8
81.5
ICD ( Wang et al., 2024 )
93.0
81.0
91.5
73.5
84.2
72.5
82.6
CODE ( Kim et al., 2024 )
92.8
81.8
91.2
72.5
84.0
72.0
82.4
SID ( Huo et al., 2025 )
90.2
75.0
86.2
65.8
83.5
73.0
79.0
Appendix
Table H: Detailed performance on CMM. Accuracy (%) on all six CMM subsets; higher is better. VL, AL, and VAL denote Visual-Language, Audio-Language, and Visual-Audio-Language, respectively. Vis., Aud., and Lang. denote Visual Dominance, Audio Dominance, and Language Dominance, respectively. Avg. is the unweighted mean over all six subsets. Bold indicates the best result within each backbone.
Method
Sem.
Temp.
Avg.
GSA
FGSA
CMSB
TA
TF
CMTB
VideoLLaMA2.1-7B-AV
Base model
50.0
67.8
71.3
50.0
30.4
50.1
53.3
VCD ( Leng et al., 2024 )
50.0
70.7
73.0
50.0
30.8
50.1
54.1
ICD ( Wang et al., 2024 )
50.0
68.2
71.3
50.0
30.1
50.1
53.3
CODE ( Kim et al., 2024 )
50.0
68.0
71.7
50.0
29.3
50.1
53.2
Appendix
Table I: Detailed performance on SVHalluc. Accuracy (%) on all six SVHalluc tasks; higher is better. GSA, FGSA, and CMSB denote Global Semantic Alignment, Fine-Grained Semantic Alignment, and Cross-Modal Semantic Binding, respectively. TA, TF, and CMTB denote Temporal Alignment, Temporal Forecasting, and Cross-Modal Temporal Binding. Avg. is the unweighted mean over all six tasks. Bold indicates the best result within each backbone, including ties.
Figure H: Evidence attenuation and recovery across input conditions. Correct-answer probability on AVHBench for audio-required and video-required questions, averaged equally across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and VideoLLaMA2.1-7B-AV. The required modality provides substantially stronger support than question-only inference; RED recovers and further strengthens this support under the full audio-visual input.
Method
VL
AL
VAL
Vis.
Aud.
Lang.
Avg.
Base model
90.5
87.0
92.2
69.2
84.2
76.5
83.3
CAD ( Shi et al., 2024 ) ( α=0.5 )
91.2
85.5
93.8
69.2
86.0
80.5
84.4
M3ID ( Favero et al., 2024 )
91.2
86.2
93.8
69.5
86.0
80.5
84.5
DCPMI ( Chen et al., 2026 ) ( β=0.25 )
93.0
85.5
94.2
70.2
86.5
77.2
84.5
RED (Ours)
92.5
84.2
96.0
75.8
87.0
81.5
86.2
Appendix
Table J: Comparison with PMI-based decoding on CMM. Accuracy (%) with Qwen2.5-Omni-3B; higher is better.
Method
Audio
Visual
Audio-Visual
Cnt.
Comp.
Cnt.
Loc.
Exist.
Loc.
Cnt.
Comp.
Temp.
Avg.
Base model
78.6
59.3
71.9
84.2
75.5
47.4
54.2
57.0
43.1
64.5
VCD ( Leng et al., 2024 )
82.4
64.8
72.2
82.0
74.9
42.8
54.0
58.8
40.5
64.5
ICD ( Wang et al., 2024 )
82.3
63.6
72.2
84.1
75.6
44.6
56.6
58.9
44.3
65.7
CODE ( Kim et al., 2024 )
80.3
63.5
70.4
83.3
74.5
44.2
48.9
58.9
42.8
63.7
SID ( Huo et al., 2025 )
84.0
61.1
72.9
80.2
72.9
42.9
51.1
54.9
36.9
62.9
Appendix
Table K: Performance on MUSIC-AVQA. Accuracy (%) with Qwen2.5-Omni-7B; higher is better. Avg. denotes accuracy over all 9,129 questions in the official corrected test split. Bold indicates the best displayed result in each column, including ties. Cnt., Comp., Loc., Exist., and Temp. denote counting, comparative, location, existential, and temporal questions, respectively.
Method
Audio
Audio-Visual
Cnt.
Comp.
Exist.
Loc.
Cnt.
Comp.
Temp.
Avg.
Base model
59.7
39.9
43.3
19.6
32.9
31.5
17.5
31.9
VCD ( Leng et al., 2024 )
62.3
41.5
45.5
18.3
33.7
31.5
17.3
32.4
ICD ( Wang et al., 2024 )
60.2
39.9
43.8
19.1
35.6
31.8
19.1
32.7
CODE ( Kim et al., 2024 )
59.7
39.4
42.7
19.9
30.0
34.7
19.1
32.3
SID ( Huo et al., 2025 )
59.7
39.4
49.7
19.1
27.5
32.8
14.7
31.5
Appendix
Table L: Performance on MUSIC-AVQA-Hard. Accuracy (%) with Qwen2.5-Omni-7B; higher is better. Avg. denotes accuracy over all 2,508 questions. Bold indicates the best displayed result in each column, including ties. Cnt., Comp., Loc., Exist., and Temp. denote counting, comparative, location, existential, and temporal questions, respectively.
Figure I: Examples of evidence attenuation and recovery with Qwen2.5-Omni-7B. Representative AVHBench cases where RED recovers question-required evidence under full audio-visual input.
Figure J: Qualitative examples with Qwen2.5-Omni-3B. Representative hallucination cases and corresponding predictions.
Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.
Xuanchen Li, Yuheng Lu, Chenrui Cui +6
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1
While Audio-Visual Language Models (AVLMs) have achieved remarkable progress over recent years, their reliability is bottlenecked by cross-modal hallucination. A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence. To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preference learning framework introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal. Extensive experiments demonstrate that ACPO establishes highly faithful audio grounding and mitigates audio hallucination.
Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing benchmarks focus on environmental sounds (e.g., dog barking) to indicate event occurrence. In contrast, human speech carries fundamentally different, rich semantics and temporal structures, yet it remains unexplored whether current models can accurately align speech content with corresponding visual signals. In this work, we show that speech content can induce hallucinations in audio-visual LLMs. To systematically study this, we introduce SVHalluc, the first comprehensive benchmark for evaluating speech-vision hallucination in audio-visual LLMs. Our benchmark diagnoses speech-vision hallucinations from two critical and complementary aspects: semantic and temporal. Experimental results demonstrate that state-of-the-art open-source audio-visual LLMs struggle with aligning speech content with corresponding visual signals, with a near-random accuracy on multiple tasks. In contrast, Gemini 2.5 Pro significantly outperforms the open-source models. Our analysis suggests that their failures stem from limited ability in cross-modality understanding, despite strong performance in single-modality perception. Our work uncovers a new and fundamental limitation of current audio-visual LLMs and highlights the need for speech-grounded video comprehension. Project page: https://chenshuang-zhang.github.io/projects/svhalluc/.
Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1