Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models

Ami Baid; Zihui Xue; Kristen Grauman

cs.CVApr 15, 2026

Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models

Authors: Ami Baid, Zihui Xue, Kristen Grauman

Organizations: University of Texas at Austin, Austin, USA

Abstract

While Audio-Visual Language Models (AVLMs) have achieved remarkable progress over recent years, their reliability is bottlenecked by cross-modal hallucination. A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence. To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preference learning framework introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal. Extensive experiments demonstrate that ACPO establishes highly faithful audio grounding and mitigates audio hallucination.

Figures & tables

Figure 1: Correcting cross-modal hallucination in AVLMs. Current models often exploit real-world co-occurrence shortcuts, leading them to hallucinate sounds based on what is seen rather than what is heard. Here, the base AVLM incorrectly predicts a siren due to the visual presence of a police car. Our approach explicitly corrects this class of modality attribution errors, decoupling the visual shortcut to accurately ground the response in the actual audio track.

Figure 2: Overview of ACPO. (a) Multimodal Data curation: each joint audio-visual caption is decomposed into modality-specific targets $y^{\text{vis}}$ and $y^{\text{aud}}$ , and audio-swapped inputs $(v_{A},a_{B})$ are constructed by replacing the original audio track with a mismatched one. (b) Preference pair construction: audio-attribution pairs (left) use the swapped input $(v_{A},a_{B})$ to penalize visually-driven responses to audio-focused prompts, preferring $y^{\text{aud}}$ over $y^{\text{vis}}$ . Audio-sensitivity pairs (right) penalize audio-invariant predictions by preferring the original audio-visual caption $y^{\text{av}}$ under the aligned input $(v_{A},a_{A})$ over the same caption under the swapped input $(v_{A},a_{B})$ .

Figure 3: Visual dominance in AVLMs is asymmetric. On the AVHBench [ 21 ] video-driven audio hallucination task (V

\rightarrow

A), adding more video frames progressively degrades performance (76.8

\rightarrow

74.0), as visual priors override auditory evidence. On the audio-driven video hallucination task (A

\rightarrow

V), adding audio improves performance (61.5

\rightarrow

77.2), indicating the model is robust to audio interference but highly susceptible to visual dominance.

Figure 4: Qualitative examples of video-driven audio hallucination. Each row shows a video clip (top), its corresponding audio waveform with labeled sound events (middle), and model responses to an audio-focused yes/no question (bottom). In all three cases, the audio contains no evidence of the queried sound, yet all baselines hallucinate affirmative responses. ACPO (Ours) correctly grounds its response in the audio signal.

Figure 5: Qualitative examples of audio-focused captioning. Each row shows a video clip with its labeled audio waveform, a reference audio caption, and model-generated audio captions. (a) The video depicts cows on a hillside, and the audio contains a man speaking and birds chirping. All baselines produce captions grounded in the visual scene (“leading cows up a hill”), failing to describe the actual audio content. ACPO correctly identifies both auditory events. (b) The video shows frogs, but the audio contains a woman speaking and a cat meowing. All baselines hallucinate a frog croaking. ACPO alone correctly describes what is heard.

	AVHBench						CMM
	Audio Hallucination				Overall		Aud-Lang			Overrely Vision			Overall
Method	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	PA $\uparrow$	HR $\uparrow$	Acc $\uparrow$	PA $\uparrow$	HR $\uparrow$	Acc $\uparrow$	Acc $\uparrow$
Gemini-Flash-1.5 ∗ [ 45 ]	57.9	94.7	71.9	63.0	77.8	73.2	88.5	39.5	64.0	79.0	36.5	57.8	76.3
Qwen2.5-Omni ∗ [ 6 ]	60.8	98.8	75.3	67.6	76.4	70.9	92.0	78.0	85.0	95.0	56.5	75.8	81.0
MiniCPM-o ∗ [ 6 ]	70.4	78.6	74.4	72.8	75.1	73.7	95.0	53.0	74.0	91.0	56.5	73.8	76.0
Base model [ 9 ]	68.6	88.5	77.3	74.0	77.6	75.6	85.5	85.5	85.5	82.0	64.5	73.3	82.5

Table 1: Correcting audio hallucination on two public benchmarks. Our method achieves the best overall performance on both benchmarks, demonstrating superior audio hallucination resistance and discrimination. PA: accuracy on yes-instances, HR: accuracy on no-instances. Best in bold. ∗ Results reported in cited work.

	Audio	Audio	Video	Video	Average
Method	(Original)	(Swap)	(Original)	(Swap)
Base model [ 9 ]	18.8/27.9	15.0/13.1	22.7/28.5	23.2 /29.7	19.9/24.8
SFT [ 36 ]	22.4/31.3	17.3/16.9	19.9/ 33.6	20.9/ 40.4	20.1/30.5
DPO [ 38 ]	22.4/27.6	18.0/16.5	22.9 /23.3	23.2/27.1	21.6/23.6
OmniDPO [ 6 ]	23.1/33.0	18.9/18.9	21.9/25.1	21.4/26.1	21.3/25.8
Ours	27.0 / 43.6	23.3 / 30.6	20.4/27.6	20.2/29.7	22.7 / 32.9

Table 2: Unimodal captioning evaluation on our newly introduced eval set. Our method achieves the largest gains on audio captioning while remaining competitive on video. Each cell shows METEOR/CIDEr (×100),

\uparrow

. Because each split uses modality-specific references, these metrics capture fidelity to the intended modality. Original: matched audio-video pairs; Swap: mismatched but plausible substitutions. Best in bold.

	Audio Halluc.		Overall
Model	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
video-SALMONN-2 (base) [ 44 ]	63.2	54.0	71.6	67.0
+ ACPO	68.8	71.6	75.0	76.5
Qwen2.5-Omni (base) [ 51 ]	66.7	74.8	68.4	74.8
+ ACPO	69.3	76.2	71.0	76.1

Table 3: ACPO transfers to architectures beyond Video-LLaMA2. Applied to video-SALMONN-2 and Qwen2.5-Omni, ACPO reduces audio hallucination and improves overall AVHBench performance on both, suggesting that its dual-axis preference signal targets a general grounding failure rather than a single-backbone artifact. Best per backbone in bold.

Configuration	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$
Ours	79.9	78.2	82.8	80.4
w/o Attribution	79.1	76.2	84.6	80.2
w/o Sensitivity	79.3	78.3	81.0	79.6

Table 4: Pair type ablation on the AVHBench audio hallucination task.

Pair Type	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$
Audio-attribution (low sim swap)	77.7	76.0	81.1	78.4
Audio-attribution (high sim swap)	77.0	74.9	81.1	77.9
Audio-attribution (no swap)	76.9	77.6	75.7	76.6
Audio-sensitivity (low sim swap) †	79.4	78.9	80.1	79.5
Audio-sensitivity (high sim swap) †	79.8	78.9	81.3	80.1

Table 5: Similarity score ablation on the AVHBench audio hallucination task. Each row trains with a single pair type in isolation. Low-similarity swaps introduce stronger audio-visual conflict; high-similarity swaps produce subtler mismatches. † Configurations where video hallucination task accuracy fell below the base model, indicating overcorrection toward audio.

Figure 6: Qualitative examples illustrating limitations. (Top) Joint audio-visual captioning: ACPO provides the most complete multimodal description but generalizes visual content in favor of auditory detail. (Bottom) Audio-focused question answering: all methods fail to detect a brief, subtle coin clinking sound.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	AVHBench						CMM
	Video Hallucination				AV Cap		VAL			Overrely Audio
Method	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	M $\uparrow$	C $\uparrow$	PA $\uparrow$	HR $\uparrow$	Acc $\uparrow$	PA $\uparrow$	HR $\uparrow$	Acc $\uparrow$
Base model [ 9 ]	75.4	80.6	78.0	77.2	17.1	18.3	83.5	96.0	89.8	85.5	84.5	85.0
SFT [ 36 ]	76.2	81.0	78.5	77.9	17.1	18.1	83.5	97.0	90.3	86.0	78.0	82.0
DPO [ 38 ]	75.3	81.6	78.3	77.5	17.2	18.5	84.5	96.5	90.5	86.0	80.5	83.3
OmniDPO [ 6 ]	79.3	77.5	78.4	78.7	16.8	16.1	79.5	98.5	89.0	81.0	85.5	83.3

Appendix

Table 6: Additional results on AVHBench and CMM. ACPO remains competitive with other baselines. AV Cap: audio-visual captioning (M: METEOR, C: CIDEr), VAL: vision-audio-language. PA: accuracy on yes-instances, HR: accuracy on no-instances. Best in bold.

	Audio Halluc.		Video Halluc.
Model	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Base model [ 9 ]	74.0	77.3	77.2	78.0
+ SFT (audio swaps) [ 36 ]	74.0	74.9	64.3	66.7

Appendix

Table 7: Training with SFT on audio-swapped data, without preference optimization, hurts video hallucination while leaving audio hallucination roughly unchanged.

Preference Optimization

Preference optimization aligns AI systems with human or task-specific judgments by training them to favor better-ranked outputs.

See all 154 papers

When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation
Oct 2, 2026Hyunjae Ra, Aecheon Jung, Jungin Park +1Multimodal Hallucination Hallucination Mitigation
Sungkyunkwan University · Daegu Gyeongbuk Institute of Science and Technology (DGIST)
Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering
Oct 14, 2025Tsung-En Lin, Kuan-Yi Lee, Hung-Yi LeeAudio QA Hallucination in Language Models
National Taiwan University, Taipei, Taiwan · ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan

May 13, 2026J/K move · Enter open · S save

When Vision Speaks for Sound

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai+5

dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison+1

Audio-Language Model Evaluation Audio-Visual Understanding Video-Language Model Evaluation Audio Reasoning Audio Understanding Video-Language Models+5

Open paper AskPDF arXiv

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Abstract

Preference Optimization

Explore similar work

When Vision Speaks for Sound

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering