Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization

Bebe Cosgrove; Aaron Isidore Grace; Weiran Wang

cs.SDOct 2, 2026

Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization

Authors: Bebe Cosgrove, Aaron Isidore Grace, Weiran Wang

Organizations: Smith College, Northampton, USA · University of Iowa, Iowa City, USA

Abstract

Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucinations, we propose utilizing the multimodal Direct Preference Optimization (mDPO) objective, which forces the model to ground its generation in the acoustic input by contrasting intact and distorted audio counterparts. We extend this preference learning framework to the audio domain by applying a variety of acoustic perturbations. Evaluating the Qwen2-Audio backbone across the DCASE 2025 Challenge and AH Existence datasets, we demonstrate that extending mDPO to LALMs significantly enhances temporal reasoning in the complex DCASE dataset, and improves performance on basic existence verification in the AH benchmark. We identify temporal reversal, frequency masking, and random noise as the most effective perturbations. Ultimately, our approach achieves an absolute accuracy improvement of 14.0% on the DCASE 2025 dataset and 27.4% on AH Existence.

Figures & tables

Figure 1: Overview of the mDPO framework for audio hallucination reduction.

Method	Overall Accuracy
Baseline	56.0%
mDPO	71.4%
$\quad-\mathcal{L}_{\mathrm{DPO}}$	70.4%
$\quad-\mathcal{L}_{\mathrm{CoPO}}$	68.9%
$\quad-\mathcal{L}_{\mathrm{AncPo}}$	64.0%

Table 1: Ablation on mDPO Loss terms with no audio perturbation on DCASE 2025 validation set.

$\beta$	Overall Accuracy
0.05	64.4%
0.10	69.4%
0.15	69.7%
0.20	71.4%
0.25	70.0%

Table 2: Beta values and corresponding accuracy with no audio perturbation on DCASE 2025 validation set.

Parameters $(n_{\text{masks}},\;w_{\max})$	Overall Accuracy	Temporal Accuracy	Complex Accuracy
Baseline	56.0%	39.2%	62.3%
(1, 0.1)	69.8%	44.1%	79.3%
(2, 0.1)	69.6%	42.8%	79.5%
(2, 0.08)	69.8%	43.1%	79.6%
(3, 0.1)	69.3%	42.2%	79.4%
(3, 0.08)	70.4%	46.7%	79.2%

Table 3: Time masking hyperparameter tuning on DCASE 2025 validation set.

$\sigma_{wave}$	Overall Accuracy	Temporal Accuracy	Complex Accuracy
0	56.0%	39.2%	62.3%
0.01	69.0%	42.5%	78.8%
0.02	69.5%	43.1%	79.3%
0.03	69.2%	43.8%	78.6%
0.05	68.3%	43.8%	77.4%

Table 4: Noise perturbation hyperparameter tuning on DCASE 2025 validation set.

Perturbation	Overall Accuracy	Temporal Accuracy	Complex Accuracy
Baseline	56.0%	39.2%	62.3%
No Audio	66.7%	43.1%	75.5%
Reverse	67.1%	44.1%	75.6%
Random Noise	67.4%	43.8%	76.2%
Time Masking	66.6%	39.5%	76.7%
Frequency Masking	67.3%	43.4%	76.2%

Table 5: Qwen2 accuracy on DCASE 2025 test set across different perturbations.

Perturbation	Test Accuracy
Baseline	55.6%
No Audio	81.3%
Reverse	83.0%
Random Noise	81.2%
Frequency Masking	79.2%
Contrastive Decoding [ 8 ]	76.7%

Table 6: Qwen2 accuracy on AH Existence across different perturbations.

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 215 papers

Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering
Oct 14, 2025Tsung-En Lin, Kuan-Yi Lee, Hung-Yi LeeAudio QA Hallucination in Language Models
National Taiwan University, Taipei, Taiwan · ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan
Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination
Oct 9, 2026Mohammad Khalooei, Mohammad SabokrouAudio-Language Models Audio-Language Model Hallucination Detection
Sharif University of Technology, Iran · Amirkabir University of Technology, Iran · New Uzbekistan University, Uzbekistan +1
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Oct 14, 2025J/K move · Enter open · S save

Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering

Tsung-En Lin, Kuan-Yi Lee, Hung-Yi Lee

National Taiwan University, Taipei, Taiwan · ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan

Audio QA Hallucination in Language Models Audio Understanding Audio-Language Models Hallucination Mitigation Activation Steering

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) excel in Audio QA but often suffer from hallucinations ungrounded in the audio. To our knowledge, we are the first to propose applying vector steering to the audio domain to mitigate this. Unlike text-based steering, our silence-anchored contrastive approach steers the model away from hallucinations by contrasting active audio against a silent baseline. Probing internal states reveals a strong correlation between specific layer representations and output correctness. Leveraging this, we introduce Layer-Weighted Vector Steering (LWVS), a training-free intervention that increases steering strength at influential layers. On the Audio Hallucination QA dataset, LWVS significantly outperforms baselines, boosting Recall on the Gemma model by 15.6% (53.4% to 69.0%). Crucially, MMAU benchmark tests confirm LWVS preserves and even enhances general audio understanding, achieving an 8% relative accuracy increase on the Qwen model (54.8% to 59.2%).

Abstract

Audio Understanding

Explore similar work

Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering