EvA: An Evidence-First Audio Understanding Paradigm for LALMs

Xinyuan Xie; Shunian Chen; Zhiheng Liu; Yuhao Zhang; Zhiqiang Lv; Liyin Liang; Benyou Wang

cs.SDMar 29, 2026

EvA: An Evidence-First Audio Understanding Paradigm for LALMs

Authors: Xinyuan Xie, Shunian Chen, Zhiheng Liu, Yuhao Zhang, Zhiqiang Lv, Liyin Liang, Benyou Wang

Organizations: The Chinese University of Hong Kong, Shenzhen · Didi Chuxing

Abstract

Large Audio Language Models (LALMs) still struggle in complex acoustic scenes because they often fail to preserve task-relevant acoustic evidence before reasoning begins. We identify this error pattern as the evidence bottleneck: state-of-the-art systems show larger deficits in acoustic evidence extraction than in downstream reasoning, suggesting that upstream perception is often the limiting factor. To address this problem, we propose EvA (Evidence-First Audio), a dual-path architecture that enhances acoustic evidence preservation through hierarchical aggregation and non-compressive, time-aligned fusion. We also build EvA-Perception, a large-scale training set with about 54K event-ordered captions and 500K evidence-grounded QA pairs. Under a unified zero-shot protocol, EvA achieves the best open-source Perception results on MMAU, MMAR, and MMSU, with the largest gains on perception-heavy splits. Human evaluation on open-ended captioning further shows improved fine-grained acoustic coverage and caption quality. These results support the evidence-first hypothesis: stronger audio understanding depends on preserving acoustic evidence before reasoning. Project can be found at https://satsuki2486441738.github.io/EvA/.

Figures & tables

Figure 1: Perception–reasoning performance gap comparison between models and humans. Performance is averaged over Qwen2.5–Omni and Kimi–Audio.

Figure 2: Model architecture. The left half of the left panel shows the Kimi-Audio backbone, while the right half illustrates the additional EvA Path modules. Audio is encoded into four frequency-band features by the CED Encoder, pooled across frequencies, fused via cross-attention, temporally aligned with Whisper, and integrated through gated additive fusion.

Name	# of Audio/QA	Avg. Caps Len	Visual	Music	Speech	Integration	Temporal Info
AudioCaps ( Kim et al., 2019 )	46k/46k	9.03	✗	✗	✗	✗	✗
Clotho ( Drossos et al., 2020 )	5k/5k	11.00	✗	✗	✗	✗	✗
LAION-Audio-630K ( Wu et al., 2023 )	630k/630k	7.30	✗	✗	✗	✗	✗
WavCaps ( Mei et al., 2024 )	403k/403k	7.80	✗	✗	✗	✗	✗
AudioSetCaps ( Bai et al., 2025a )	1.9M/1.9M	28.00	✗	✗	✗	✗	✗
Auto-ACD ( Sun et al., 2024 )	1.5M/1.5M	18.10	✓	✗	✗	✓	✗

Table 1: Comparison of open-source audio caption datasets.

Model	MMAU-Clean		MMAR		MMSU		CochlScene
Model	Perc.	Reas.	Perc.	Reas.	Perc.	Reas.	CochlScene
Qwen2.5-Omni-7B	72.06	67.36	52.80	61.43	39.96	69.91	80.38
Audio-Flamingo-3	73.90	71.50	59.23	61.64	45.63	77.86	75.57
Step-Audio-2-mini	66.54	68.58	54.58	61.01	44.36	78.32	83.24
Audio-Reasoner	65.44	63.39	44.34	38.23	40.73	68.38	–
R1-AQA	72.06	62.87	48.75	50.19	41.68	71.94	76.30

Table 2: Main results under our unified zero-shot setting. For MMAU, we report the decontaminated split, denoted as MMAU-Clean, to reduce potential source-audio overlap with AudioSet-Strong. Green numbers denote absolute gains over Kimi-Audio-7B-Instruct. Definitions of perception and reasoning are given in Sec. 5.1 ; the full MMAU decontamination protocol and original-vs-cleaned results are provided in Appendix A.14 .

Figure 3: Qualitative comparison of captions generated by different models on AudioCaps examples.

Stage	Setting	Trainables	Start Ckpt	AudioCaps (CLAP)
Stage	Setting	Trainables	Start Ckpt	Cos	R@1	R@5
S0	Base Model	N/A	N/A	$14.61$	$9.27$	$24.97$
S1(1)	w/o CED path	Adapter	S0	$35.40$	$18.50$	$43.60$
S1(2)	w/o frequency pooling	Adapter & CED Agg.	S0	$35.54$	$21.24$	$49.61$
S1(3)	w/o crossing fusion	Adapter & CED Agg.	S0	$28.63$	$11.82$	$29.74$
S1(4)	Q-former	Adapter & CED Q-former	S0	$36.24$	$20.08$	$47.36$

Table 3: Ablations of the EvA fusion path. On AudioCaps , we report CLAP-based cosine similarity ( Cos ) and retrieval; on MMAU/MMAR/MMSU, we report Perception accuracy. Adapter denotes the Whisper adapter, and LLM denotes LoRA on the backbone. In Stage 2, w/o CED path is trained and evaluated without the CED branch.

Model	Label Recall	# Obj. Mentions	Obj. Acc.	Arena
Step-Audio-2-mini	58.6	191	67.5	43.2
Qwen2.5-Omni	57.9	166	78.3	55.1
Audio-Reasoner	51.9	128	85.9	64.2
Audio-Flamingo-3	50.4	125	83.2	39.7
R1-AQA	47.4	123	84.6	39.1
Kimi-Audio-7B	31.6	80	71.3	25.3

Table 4: Human evaluation summary on AudioSet-Strong captioning. Green numbers denote gains over Kimi-Audio-7B. Full results are in Appendix A.12 .

Benchmark	Subset	Base Arch.	EvA
MMAU-mini	Perception	75.23	76.16 +0.93
MMAU-mini	Reasoning	67.80	69.73 +1.93
MMAR	Perception	53.24	56.82 +3.58
MMAR	Reasoning	61.61	65.08 +3.47
MMSU	Perception	42.02	49.21 +7.19
MMSU	Reasoning	69.95	79.47 +9.52

Table 5: The performance of Qwen2.5-Omni architecture.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Stage 1 (Alignment)	Stage 2 (SFT, LoRA)
Trainable modules	CED Aggregator	CED Aggregator, Whisper adapter; LLM via LoRA
Dataset	EvA-Alignment	EvA-Perception
Epochs	5	2
Per-device batch	2	2
Grad. accumulation	8	16
Global batch (8 $\times$ A100)	128	256

Appendix

Table 6: Training hyperparameters. Encoders (Whisper, CED-Base) are frozen throughout.

Item	Result
Sample size	500 captions
Sampling seed	42
Audit input	Generated caption only
Flagged captions	7 / 500
Estimated leakage rate	1.4%

Appendix

Table 7: Summary of the visual-leakage audit.

Model	Label Recall	# Obj. Mentions	Obj. Acc.	Arena Shown	W/L/T	Arena
Step-Audio-2-mini	78 / 133 (58.6)	191	129 / 191 (67.5)	154	50 / 71 / 33	43.2
Qwen2.5-Omni	77 / 133 (57.9)	166	130 / 166 (78.3)	138	63 / 49 / 26	55.1
Audio-Reasoner	69 / 133 (51.9)	128	110 / 128 (85.9)	169	94 / 46 / 29	64.2
Audio-Flamingo-3	67 / 133 (50.4)	125	104 / 125 (83.2)	165	46 / 80 / 39	39.7
R1-AQA	63 / 133 (47.4)	123	104 / 123 (84.6)	137	36 / 66 / 35	39.1
Kimi-Audio-7B	42 / 133 (31.6)	80	57 / 80 (71.3)	162	23 / 103 / 36	25.3

Appendix

Table 8: Detailed human evaluation on AudioSet-Strong captioning. Values in parentheses are percentages; Arena treats ties as half a win.

MMAU split	Original	Removed	Cleaned
Perception	323	51	272
Reasoning	677	98	579
Total	1,000	149	851

Appendix

Table 9: Statistics of the decontaminated MMAU-Clean split.

Model	Original	MMAU-Clean	$\Delta$
EvA	78.33	77.57	$-0.75$
Audio-Flamingo-3	76.78	73.90	$-2.88$
Qwen2.5-Omni-7B	73.68	72.06	$-1.63$
R1-AQA	73.99	72.06	$-1.93$
Step-Audio-2-mini	70.28	66.54	$-3.73$
Kimi-Audio-7B-Instruct	67.18	66.18	$-1.01$

Appendix

Table 10: Original and decontaminated results on the MMAU Perception subset.

Model	Original	MMAU-Clean	$\Delta$
Audio-Flamingo-3	74.15	71.50	$-2.65$
Step-Audio-2-mini	71.79	68.58	$-3.21$
Qwen2.5-Omni-7B	69.72	67.36	$-2.36$
EvA	67.35	64.17	$-3.19$
R1-AQA	65.88	62.87	$-3.01$
Audio-Reasoner	65.73	63.39	$-2.35$

Appendix

Table 11: Original and decontaminated results on the MMAU Reasoning subset.

Panel of Figure 4 — Figure 4: Instruction for Caption Generation.

Panel of Figure 5 — Figure 5: Instruction for QA Generation.

Dataset	Constituent Sources	Modality	Quantity
EvA-Alignment	EvA-Captions	Sound, Speech, Music	53,934
	AudioTime ( Xie et al., 2025a )	Sound	5,000
	CommonVoice ( Ardila et al., 2019 )	Speech	20,000
	MusicBench ( Melechovsky et al., 2023 )	Music	30,000
	MusicCaps ( Agostinelli et al., 2023 )	Music	4,852
Total			113,786

Appendix

Table 12: Composition of EvA-Perception and EvA-Alignment datasets.

Panel of Figure 6 — Figure 6: Example from EvA-Captions and EvA-QA (Case 1).

Panel of Figure 7 — Figure 7: Example from EvA-Captions and EvA-QA (Case 2).

Model	Sound	Speech	Music	Avg.
Qwen2-Audio	58.86	$47.75$	$44.31$	$50.30$
Qwen2.5-Omni	73.87	$65.47$	$67.96$	$69.10$
Kimi-Audio	74.77	$62.35$	$64.24$	$67.19$
Audio-Reasoner	65.77	$66.07$	$66.77$	$65.00$
R1-AQA	74.47	$65.17$	$66.77$	$68.80$
EvA(Ours)	80.78	68.47	74.65	74.63

Appendix

Table 13: Main results on MMAU-mini-test.

Model	Single Modality			Mixed Modality				Avg.
Model	Sound	Speech	Music	S-M	S-S	M-S	S-M-S	Avg.
Qwen2-Audio	$52.73$	$42.86$	$34.95$	$36.36$	$50.46$	$45.12$	$50.00$	$44.80$
Qwen2.5-Omni	$59.39$	$61.22$	$48.06$	$54.55$	$61.01$	64.63	$58.33$	$58.30$
Kimi-Audio	$55.76$	$59.86$	$45.15$	$36.36$	$61.01$	$54.88$	$45.83$	$55.40$
Audio-Reasoner	$50.30$	$49.66$	$38.35$	$36.36$	$56.42$	$48.78$	$50.00$	$48.70$
R1-AQA	60.00	$51.36$	$42.23$	$54.55$	$57.80$	$52.44$	$45.83$	$52.30$

Appendix

Table 14: Main results on MMAR.

Setting	MMAU	MMAR	MMSU	CochlScene
Mask 0–2 kHz	$72.20$	$55.53$	$63.72$	$60.67$
Mask 2–4 kHz	$72.30$	$55.63$	$63.72$	$60.91$
Mask 4–6 kHz	$72.30$	$56.24$	$63.82$	$60.61$
Mask 6–8 kHz	$72.40$	$55.33$	$63.88$	$61.00$
Mask 0–4 kHz	$72.30$	$54.93$	$63.11$	$60.67$
Mask 4–8 kHz	$71.90$	$55.43$	$63.98$	$60.78$

Appendix

Table 15: Main results on various audio benchmarks. All numbers are average scores.

Figure 8: Comparison of audio token sequence length after feature fusion. SALMONN’s Q-Former compresses audio tokens into a shorter latent sequence, while EvA preserves sequence length via inject-and-add fusion.

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 216 papers

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Mar 19, 2026Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13Audio Understanding Audio-Language Model Evaluation
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA
Multi-Source Evidence Fusion for Audio Question Answering
Mar 18, 2026Aivo Olev, Tanel AlumäeAudio QA Audio Reasoning
Tallinn University of Technology, Estonia

Mar 19, 2026J/K move · Enter open · S save

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang, Zhehuai Chen+12

National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan

Audio Understanding Audio-Language Model Evaluation LLM Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.

Abstract

Audio Understanding

Explore similar work

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Multi-Source Evidence Fusion for Audio Question Answering