The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models

Heinrich Dinkel; Jiahao Zhou; Guanbo Wang; Yadong Niu; Junbo Zhang; Yufeng Hao; Ying Liu; Ke Li

cs.SDMar 24, 2026

The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models

Authors: Heinrich Dinkel, Jiahao Zhou, Guanbo Wang, Yadong Niu, Junbo Zhang, Yufeng Hao, Ying Liu, Ke Li,

Organizations: MiLM Plus, Xiaomi Inc., Beijing, China · DataoceanAI Inc., USA · Centre for Vision Speech and Signal Processing, University of Surrey, Guildford, UK · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Abstract

This paper presents the Interspeech 2026 Audio Encoder Capability Challenge, a benchmark specifically designed to evaluate and advance the performance of pre-trained audio encoders as front-end modules for Large Audio Language Models (LALMs). While LALMs have shown remarkable understanding of complex acoustic scenes, their performance depends on the semantic richness of the underlying audio encoder representations. This challenge addresses the integration gap by providing a unified generative evaluation framework, XARES-LLM, which assesses submitted encoders across a diverse suite of downstream classification and generation tasks. By decoupling encoder development from LLM fine-tuning, the challenge establishes a standardized protocol for general-purpose audio representations that can effectively be used for the next generation of multimodal language models.

Figures & tables

Figure 1: The XARES-LLM framework. Participants provide a frozen encoder, and the framework trains an LLM.

Track	Objective	LLM Decoder
A	Classification	Dedicated
A (Hidden)	Classification	Dedicated
B	Understanding	Dedicated
B (Hidden)	Understanding	Shared with Track B

Table 1: Overview of Challenge Tasks and Model Training

ID	Dataset	Task Type	Metric	#
$P_{1}$	ASV2015 [ 5 ]	Spoofing	Acc	2
$P_{2}$	CREMA-D [ 6 ]	Emotion	Acc	5
$P_{4}$	Fluent Spch Comm [ 7 ]	Intent	Acc	248
$P_{5}$	LibriSpeech [ 8 ]	Gender	Acc	2
$P_{9}$	LibriCount [ 9 ]	Speaker count	Acc	11
$P_{11}$	Speech Commands [ 10 ]	Keyword spotting	Acc	30

Table 2: Track A (Classification) datasets in XARES-LLM, including mapping to Leaderboard IDs (

P_{n},H_{n}

). Domains: blue (speech), yellow (sound), and red (music).

Hidden Evaluation Datasets
ID	Dataset	Task Type	Metric
$U_{1}$	AISHELL-1-100h [ 23 ]	ASR	iCER
$U_{2}$	Clotho [ 24 ]	Sound Caption	FENSE
$U_{3}$	LibriSpeech-100h [ 8 ]	ASR	iWER
$U_{4}$	MECAT [ 25 ]	General Caption	DATE
$U_{5}$	The Song Describer Dataset [ 26 ]	Music Caption	FENSE

Table 3: Track B (Understanding) datasets in XARES-LLM, including hidden evaluation tasks. Public versions are hosted on Huggingface. Domains: blue (speech), yellow (sound), and red (music).

Team	Avg.	Public Test Results ( $P_{n}$ )															Hidden Test Results ( $H_{n}$ )
Team	Avg.	$P_{1}$	$P_{2}$	$P_{3}$	$P_{4}$	$P_{5}$	$P_{6}$	$P_{7}$	$P_{8}$	$P_{9}$	$P_{10}$	$P_{11}$	$P_{12}$	$P_{13}$	$P_{14}$	$P_{15}$	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	$H_{5}$	$H_{6}$
THUVoice	91.2	99.0	87.9	91.3	99.4	95.9	32.1	88.4	90.9	47.9	78.8	93.5	88.9	93.9	97.2	93.9	87.5	99.9	98.5	98.1	100.0	99.0
THU-HSCI-2	90.8	98.6	86.0	92.3	99.4	89.1	33.9	89.1	93.9	50.6	78.7	82.6	89.6	94.5	97.7	87.3	87.5	99.7	99.5	98.8	100.0	83.0
trans-encoder	90.0	98.3	87.3	92.3	99.6	73.1	32.4	87.6	92.9	50.9	75.6	84.6	88.1	93.9	98.5	94.1	87.5	95.7	99.5	98.4	100.0	98.7
IASP Lab	89.4	99.0	85.2	91.5	99.2	71.6	30.9	85.9	89.9	49.3	75.3	81.0	84.8	94.0	98.3	97.5	86.4	99.4	99.5	97.4	100.0	96.7
MIT SLS	88.4	96.7	69.0	91.3	99.5	64.4	27.5	84.2	90.9	43.6	72.8	79.1	88.1	93.7	97.1	95.5	87.5	99.8	99.0	98.0	100.0	98.7

Table 4: Track A leader leaderboard results. Public (

P

) and Hidden (

H

) metrics are provided.

Team	Strategy	Pre-trained Encoders	Track A Rank	Track B Rank
THU-Voice	Mixture-of-Experts (MoE)	Audio-Flamingo 3, Qwen2-Audio	1	4
THU-HSCI-2	Dual-Model Fusion	Qwen2-Audio, Audio-Flamingo 3	2	3
trans-encoder	Hierarchical Extraction	StepAudio 2	3	1
IASP Lab	Dual-Branch Time Fusion	Qwen2-Audio, Whisper	4	10
EncodexPOL	Audio-text alignment	Audio-Flamingo 3, Whisper	6	5
NTU_BIO	Gated Feature Fusion	Qwen3-ASR	15	14

Table 6: Challenge submissions, their approach and (if applicable) the utilized pre-trained encoders. Methods utilizing audio encoders trained with a LALM are highlighted in grey .

Figure 2: Comparison of team rankings across Track A and Track B.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Sep 30, 2026Shengbo Cai, Zhisheng Zhang, Zichao Nie +3Audio Representation Learning Mixture of Experts
Tsinghua University, China
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model Evaluation Audio Representation Learning
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1
Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models
Sep 16, 2026Pei-Jun Liao, Hung-Shin Lee, Wenze Ren +3Audio-Language Model Evaluation Audio-Language Models
Inst. of Information Science, Academia Sinica, Taiwan · Grad. Inst. of Electrical Engineering, National Taiwan University, Taiwan · Grad. Inst. of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan +3

Sep 30, 2026J/K move · Enter open · S save

UniAE-MoE: A Unified Audio Encoder via Mixture of Experts

Shengbo Cai, Zhisheng Zhang, Zichao Nie, Jing Peng+2

Tsinghua University, China

Audio Representation Learning Mixture of Experts Audio Understanding

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE's understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of UniAE-MoE for unified audio understanding across speech, music, and general audio domains.

Abstract

Audio-Language Model Evaluation

Explore similar work

UniAE-MoE: A Unified Audio Encoder via Mixture of Experts

Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models