Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Hyunjae Ra; Aecheon Jung; Jungin Park; Sungeun Hong

cs.AIOct 2, 2026

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Authors: Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong

Organizations: Sungkyunkwan University · Daegu Gyeongbuk Institute of Science and Technology (DGIST)

Abstract

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.

Figures & tables

Figure 1: Motivation of this study. (a) A correct prediction does not necessarily indicate successful hallucination mitigation. (b) When question-required evidence changes, the prediction should change accordingly. (c) When non-required evidence changes, it should remain unchanged. The base model violates both behaviors, whereas RED follows the evidence required by the question.

Figure 2: Overview of RED for an audio-required case. (i) Given a query, RED first identifies the evidence required to answer it, selecting among none, audio, video, audio + video, or audio-visual interaction. (ii) The frozen AV-LLM produces the audio-visual prediction and only the auxiliary predictions required by the selected evidence type, from which RED measures audio, video, and non-additive audio-visual contributions relative to the textual reference. (iii) The selected contribution augments the original audio-visual prediction to produce the final evidence-guided prediction.

Figure 3: Evidence following under modality interventions. Higher sensitivity and lower invariance violation indicate better evidence following. The dashed reference uses only the modality required by the question; its invariance violation is zero by construction because the intervened non-required modality is not used. Base receives both audio and video. Results are averaged over question-required evidence types and three backbones.

Method	CMM				AVHBench				SVHalluc
Method	Vis.	Aud.	Lang.	Avg.	ADV	VDA	AVM	Avg.	Sem.	Temp.	Avg.
VideoLLaMA2.1-7B-AV
Base	73.5	84.0	72.5	82.8	80.4	76.5	57.8	71.6	63.0	43.5	53.3
VCD ( Leng et al., 2024 )	70.8	84.5	73.8	81.5	80.7	74.6	56.3	70.6	64.6	43.6	54.1
ICD ( Wang et al., 2024 )	73.5	84.2	72.5	82.6	80.7	77.0	57.8	71.8	63.2	43.4	53.3
CODE ( Kim et al., 2024 )	72.5	84.0	72.0	82.4	80.5	76.0	57.8	71.4	63.2	43.1	53.2

Table 1: Performance on audio–visual hallucination benchmarks. Accuracy (%) on CMM, AVHBench, and SVHalluc. Vis., Aud., and Lang. denote the CMM visual-, audio-, and language-dominance subsets. For SVHalluc, Sem. and Temp. average the semantic (GSA, FGSA, CMSB) and temporal (TA, TF, CMTB) tasks. Avg. is the unweighted mean over all tasks in each benchmark.

Q-guided	PMI	CMM		AVH
Q-guided	PMI	Acc.	AUC	Acc.	AUC
✗	✗	84.8	92.3	75.2	82.0
✓	✗	86.8	93.6	76.2	82.7
✗	✓	84.2	91.9	75.8	82.8
✓	✓	87.0	94.0	78.6	86.1

Table 3: Component analysis of evidence correction. We ablate question-guided evidence selection and PMI-based contribution decomposition with the remaining RED procedure fixed. Results are averaged over three backbones. AUC denotes AUROC.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A: Question-guided evidence selection. The frozen backbone identifies the required evidence using the question text alone.

Method	Centering	Acc.	AUROC
VideoLLaMA2.1-7B-AV
RED (Ours)	$\times$	85.5	92.5
RED (Ours)	$\checkmark$	85.6	92.8
Qwen2.5-Omni-3B
RED (Ours)	$\times$	86.1	93.6
RED (Ours)	$\checkmark$	86.2	93.6

Appendix

Table A: Effect of score centering on CMM. Task-macro accuracy and AUROC (%) for RED with and without centering.

Figure B: Sensitivity to centering sample size on CMM. We vary the number of unlabeled centering questions under a fixed sampling configuration. Dashed lines denote RED without centering; the dotted vertical line marks the default budget of 100 questions.

Figure C: Robustness to centering sample selection on CMM. Small circles show results from five random draws of 100 unlabeled questions. Diamonds and error bars indicate the mean and one sample standard deviation. Stars show the reported RED results, and squares denote RED without centering.

Figure D: Effect of choosing different evidence contributions. We keep the evidence selection and centering fixed and vary only the contribution added to the audio-visual score. The left panel includes all eligible groups, while the right panel focuses on Audio and Video groups and additionally compares adding the other unimodal contribution. No correction denotes the centered audio-visual score without an additional evidence contribution. Higher TPR and TNR are better.

Task	Evidence	$+\Delta^{A}$		$+\Delta^{V}$		$+\Delta^{I}$
Task	Evidence	Acc.	AUROC	Acc.	AUROC	Acc.	AUROC
VDA	$A$	81.27	89.24	70.39	76.89	46.38	44.60
ADV	$V$	77.99	84.26	83.63	88.82	64.08	68.20
AVM	$A\leftrightarrow V$	69.56	77.65	62.42	68.51	70.68	78.54

Appendix

Table B: Modality-specific and residual contributions on AVHBench. Accuracy and AUROC (%) with Qwen2.5-Omni-3B when adding the audio (

\Delta^{A}

), video (

\Delta^{V}

), or residual audio-visual (

\Delta^{I}

) contribution to the same centered audio-visual score. VDA, ADV, and AVM correspond to Audio (

A

), Video (

V

), and Relation (

A\leftrightarrow V

) evidence requirements, respectively. The centering offsets are fixed across variants. Bold indicates the best result in each row for each metric.

$A+V$ subset	$\mathbf{n}$	$+\Delta^{A}$	$+\Delta^{V}$	$+(\Delta^{A}+\Delta^{V})$	$+\Delta^{I}$
Prompt-selected	1,786	36.73	38.63	38.19	33.82
Oracle-tagged	2,124	37.95	38.79	40.16	30.23

Appendix

Table C: Additive contributions for both-modality evidence on MUSIC-AVQA-Hard. Accuracy (%) with Qwen2.5-Omni-7B. Prompt-selected contains questions assigned to

A+V

by RED ’s question-guided evidence selection, whereas Oracle-tagged contains questions identified by the dataset’s Audio-Visual evidence tags. All variants use the same centering offsets and differ only in the contribution added to the centered audio-visual score. Bold indicates the best result in each row.

Figure E: Question-guided evidence selection across five benchmarks. For each backbone, proportions are computed over all questions in each evaluation split and then averaged equally across three backbones. Centers report the number of questions per backbone. Audio, Video, Both, Relation, and None select

\Delta^{A}

\Delta^{V}

\Delta^{A}+\Delta^{V}

\Delta^{I}

, and no additional correction, respectively. Both selects additive audio and video contributions; Relation selects their residual interaction. The distributions reflect question-only selector predictions, not oracle annotations. Nonzero categories below 1% are listed beneath each chart. MUSIC-AVQA uses the official corrected test annotations (9,129 questions); MUSIC-AVQA-Hard contains 2,508 questions from this split.

Stage	Time (ms)
Evidence selection	89.1
Media preparation	2372.1
Perceptual encoding	1881.4
Answer-prompt prefill	1381.7
Answer-token prediction	28.4
Score composition and overhead	37.2

Appendix

Table D: Runtime breakdown of RED . Stage times are averaged equally over CMM, AVHBench, and SVHalluc across three backbones using separate profiling runs.

Method	VideoLLaMA2.1-7B-AV			Qwen2.5-Omni-3B			Qwen2.5-Omni-7B
Method	CMM	AVH	SVH	CMM	AVH	SVH	CMM	AVH	SVH
Base	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$
VCD ( Leng et al., 2024 )	$2.2\times$	$3.0\times$	$1.6\times$	$2.9\times$	$3.2\times$	$2.9\times$	$3.0\times$	$3.3\times$	$3.0\times$
ICD ( Wang et al., 2024 )	$2.1\times$	$2.8\times$	$1.6\times$	$3.8\times$	$4.1\times$	$3.0\times$	$3.7\times$	$4.1\times$	$3.2\times$
CODE ( Kim et al., 2024 )	$2.7\times$	$3.8\times$	$2.0\times$	$3.6\times$	$3.8\times$	$2.1\times$	$4.0\times$	$3.8\times$	$2.2\times$
SID ( Huo et al., 2025 )	$1.2\times$	$1.4\times$	$1.1\times$	$1.5\times$	$1.5\times$	$1.5\times$	$1.4\times$	$1.4\times$	$1.5\times$

Appendix

Table E: Relative TTFT across audio-visual hallucination benchmarks. Time to first token normalized to the corresponding Base (

\times

Base; lower is better).

Method	VL	AL	VAL	Vis.	Aud.	Lang.	Avg.	TTFT (s)	Relative
Base model	90.50	84.75	89.50	69.25	84.50	75.25	82.29	2.717	$1.000\times$
SID (0% removed)	90.50	84.75	89.50	69.25	84.50	75.25	82.29	4.379	$1.612\times$
SID (60% removed)	87.50	83.75	88.25	71.25	81.25	77.75	81.63	4.179	$1.538\times$
SID (90% removed)	89.25	84.25	91.00	71.25	74.50	79.00	81.54	4.101	$1.509\times$
RED (Ours)	88.25	84.75	95.00	76.25	88.50	77.25	85.00	3.893	$\mathbf{1.433}\times$

Appendix

Table F: Accuracy–TTFT trade-off of SID ( Huo et al., 2025 ) pruning on CMM. Results with Qwen2.5-Omni-3B under a common 2-fps configuration. Accuracy (%) is reported on all 2,400 questions, and TTFT is normalized to the paired Base. Bold indicates the best accuracy or lowest non-base TTFT.

Test	Req.	Swap	Gold	Donor	Pairs	Samples
Sens.	Audio	A	Yes $\rightarrow$ No	Annotated	804	804
			No $\rightarrow$ Yes	Annotated	524	524
	Video	V	Yes $\rightarrow$ No	Annotated	192	192
			No $\rightarrow$ Yes	Annotated	101	101
Inv.	Video	A	unchanged	Random	674	674
	Audio	V	unchanged	Random	1,064	1,064

Appendix

Table G: Construction of evidence interventions. Required-modality swaps change the answer, whereas irrelevant-modality swaps preserve it. Pairs denote retained interventions and Samples denote unique source video–question items; the total is deduplicated.

Figure F: Illustration of pair-level evidence intervention. For a video-required question, we replace the video of AVHBench sample 01161 with that of sample 00640 while retaining the original audio from 01161. The reference answer changes accordingly, and the pair succeeds only when both the original and intervened inputs are answered correctly. This is an illustrative pair, not a retained evaluation pair; waveforms are schematic.

Figure G: Evidence following under audio-visual interventions on AVHBench. Sensitivity requires both answers in an answer-changing pair to be correct, while invariance violation measures prediction changes under non-required-modality swaps. Bars and whiskers denote the mean and min–max range across three backbones.

Method	VL	AL	VAL	Vis.	Aud.	Lang.	Avg.
VideoLLaMA2.1-7B-AV
Base model	93.5	81.2	92.2	73.5	84.0	72.5	82.8
VCD ( Leng et al., 2024 )	92.5	77.5	89.8	70.8	84.5	73.8	81.5
ICD ( Wang et al., 2024 )	93.0	81.0	91.5	73.5	84.2	72.5	82.6
CODE ( Kim et al., 2024 )	92.8	81.8	91.2	72.5	84.0	72.0	82.4
SID ( Huo et al., 2025 )	90.2	75.0	86.2	65.8	83.5	73.0	79.0

Appendix

Table H: Detailed performance on CMM. Accuracy (%) on all six CMM subsets; higher is better. VL, AL, and VAL denote Visual-Language, Audio-Language, and Visual-Audio-Language, respectively. Vis., Aud., and Lang. denote Visual Dominance, Audio Dominance, and Language Dominance, respectively. Avg. is the unweighted mean over all six subsets. Bold indicates the best result within each backbone.

Method	Sem.			Temp.			Avg.
Method	GSA	FGSA	CMSB	TA	TF	CMTB
VideoLLaMA2.1-7B-AV
Base model	50.0	67.8	71.3	50.0	30.4	50.1	53.3
VCD ( Leng et al., 2024 )	50.0	70.7	73.0	50.0	30.8	50.1	54.1
ICD ( Wang et al., 2024 )	50.0	68.2	71.3	50.0	30.1	50.1	53.3
CODE ( Kim et al., 2024 )	50.0	68.0	71.7	50.0	29.3	50.1	53.2

Appendix

Table I: Detailed performance on SVHalluc. Accuracy (%) on all six SVHalluc tasks; higher is better. GSA, FGSA, and CMSB denote Global Semantic Alignment, Fine-Grained Semantic Alignment, and Cross-Modal Semantic Binding, respectively. TA, TF, and CMTB denote Temporal Alignment, Temporal Forecasting, and Cross-Modal Temporal Binding. Avg. is the unweighted mean over all six tasks. Bold indicates the best result within each backbone, including ties.

Figure H: Evidence attenuation and recovery across input conditions. Correct-answer probability on AVHBench for audio-required and video-required questions, averaged equally across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and VideoLLaMA2.1-7B-AV. The required modality provides substantially stronger support than question-only inference; RED recovers and further strengthens this support under the full audio-visual input.

Method	VL	AL	VAL	Vis.	Aud.	Lang.	Avg.
Base model	90.5	87.0	92.2	69.2	84.2	76.5	83.3
CAD ( Shi et al., 2024 ) ( $\alpha=0.5$ )	91.2	85.5	93.8	69.2	86.0	80.5	84.4
M3ID ( Favero et al., 2024 )	91.2	86.2	93.8	69.5	86.0	80.5	84.5
DCPMI ( Chen et al., 2026 ) ( $\beta=0.25$ )	93.0	85.5	94.2	70.2	86.5	77.2	84.5
RED (Ours)	92.5	84.2	96.0	75.8	87.0	81.5	86.2

Appendix

Table J: Comparison with PMI-based decoding on CMM. Accuracy (%) with Qwen2.5-Omni-3B; higher is better.

Method	Audio		Visual		Audio-Visual
Method	Cnt.	Comp.	Cnt.	Loc.	Exist.	Loc.	Cnt.	Comp.	Temp.	Avg.
Base model	78.6	59.3	71.9	84.2	75.5	47.4	54.2	57.0	43.1	64.5
VCD ( Leng et al., 2024 )	82.4	64.8	72.2	82.0	74.9	42.8	54.0	58.8	40.5	64.5
ICD ( Wang et al., 2024 )	82.3	63.6	72.2	84.1	75.6	44.6	56.6	58.9	44.3	65.7
CODE ( Kim et al., 2024 )	80.3	63.5	70.4	83.3	74.5	44.2	48.9	58.9	42.8	63.7
SID ( Huo et al., 2025 )	84.0	61.1	72.9	80.2	72.9	42.9	51.1	54.9	36.9	62.9

Appendix

Table K: Performance on MUSIC-AVQA. Accuracy (%) with Qwen2.5-Omni-7B; higher is better. Avg. denotes accuracy over all 9,129 questions in the official corrected test split. Bold indicates the best displayed result in each column, including ties. Cnt., Comp., Loc., Exist., and Temp. denote counting, comparative, location, existential, and temporal questions, respectively.

Method	Audio		Audio-Visual
Method	Cnt.	Comp.	Exist.	Loc.	Cnt.	Comp.	Temp.	Avg.
Base model	59.7	39.9	43.3	19.6	32.9	31.5	17.5	31.9
VCD ( Leng et al., 2024 )	62.3	41.5	45.5	18.3	33.7	31.5	17.3	32.4
ICD ( Wang et al., 2024 )	60.2	39.9	43.8	19.1	35.6	31.8	19.1	32.7
CODE ( Kim et al., 2024 )	59.7	39.4	42.7	19.9	30.0	34.7	19.1	32.3
SID ( Huo et al., 2025 )	59.7	39.4	49.7	19.1	27.5	32.8	14.7	31.5

Appendix

Table L: Performance on MUSIC-AVQA-Hard. Accuracy (%) with Qwen2.5-Omni-7B; higher is better. Avg. denotes accuracy over all 2,508 questions. Bold indicates the best displayed result in each column, including ties. Cnt., Comp., Loc., Exist., and Temp. denote counting, comparative, location, existential, and temporal questions, respectively.

Panel of Figure 29 — Figure I: Examples of evidence attenuation and recovery with Qwen2.5-Omni-7B. Representative AVHBench cases where RED recovers question-required evidence under full audio-visual input.

Panel of Figure 30 — Figure J: Qualitative examples with Qwen2.5-Omni-3B. Representative hallucination cases and corresponding predictions.

Multimodal Hallucination

Multimodal hallucination occurs when AI systems produce claims unsupported by, or conflicting with, image, audio, video, or other input evidence.

See all 51 papers

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
May 11, 2026Xuanchen Li, Yuheng Lu, Chenrui Cui +6Cross-Modal Learning Multimodal QA
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1
Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models
Apr 15, 2026Ami Baid, Zihui Xue, Kristen GraumanPreference Optimization Multimodal Contrastive Learning
University of Texas at Austin, Austin, USA
SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
May 31, 2026Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1Audio-Visual Understanding Multimodal Large Language Models
KAIST

May 11, 2026J/K move · Enter open · S save

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang+5

Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China+1

Cross-Modal Learning Multimodal QA Multimodal Hallucination Audio-Visual QA Multimodal CoT Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.

Abstract

Multimodal Hallucination

Explore similar work

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models