Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge

Yushi Wang; Zeyan Song; Xiaobin Rong; Chengwei Liu; Jun Gao; Jing Lu

eess.ASOct 9, 2026

Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge

Authors: Yushi Wang, Zeyan Song, Xiaobin Rong, Chengwei Liu, Jun Gao, Jing Lu

Organizations: Key Laboratory of Modern Acoustics, Nanjing University, Nanjing, China · NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Beijing, China · Qwen Business Unit of Alibaba, Hangzhou, China

Abstract

This paper describes our systems for the Track 1 time-stamped speaker-attributed ASR task of the SmartGlasses Challenge. To handle overlapping speech, background noise and far-field conditions in two-person conversations, we employ SoulX-Transcriber, a multimodal large language model that takes full-length audio as input and generates structured transcriptions. In addition, we investigate the effect of speech frontend processing, including a rule-based Guided Source Separation method and a data-driven speech separation model, both of which improve performance on the validation set. Considering inference stability, our final submission uses the fine-tuned SoulX-Transcriber system, achieving a tcpCER of 6.22 on the test set.

Figures & tables

Fig. 1: Overview of our systems, where the three systems are developed and evaluated independently.

System	tcpCER	cpCER	DER
Baseline VibeVoice-ASR	14.09	12.88	17.24
System 1 SoulX-pretrained	9.02	8.45	3.70
System 1 SoulX-finetuned	7.82	7.30	2.56
System 2 GSS-SoulX
w/ oracle VAD	7.39	6.65	1.14
w/ estimated VAD	6.97	6.12	2.62

TABLE I: Experimental results on the Track 1 TSA-ASR task official validation set.

Fig. 2: Chat-level average tcpCER across different speech overlap ratio levels (Low/Mid/High) for different systems on the Track 1 TSA-ASR task official validation set.

Speaker-Attributed ASR

See all 11 papers

The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models
Aug 12, 2026Dehui Gao, Zhixian Zhao, Zhennan Lin +14Audio-Language Model Evaluation ASR Evaluation
Huawei · AIShell Inc · Shanghai Jiao Tong University +4
Phonology-Aware Speaker Extraction Frontend for Audio LLMs
Oct 9, 2026Yanxiang Chen, Haoyuan Deng, Jinyu Wang +4Target Speaker Extraction Speech Processing
Sun Yat-Sen University, Zhuhai, China · Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Fudan University, Shanghai, China
G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition
Mar 11, 2026Jing Peng, Ziyi Chen, Haoyu Li +7Speaker-Attributed ASR Automatic Speech Recognition
Shanghai Jiao Tong University · Shenzhen Research Institute of Big Data · Nanjing University +2

Aug 12, 2026J/K move · Enter open · S save

The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models

Dehui Gao, Zhixian Zhao, Zhennan Lin, Yujie Liao+13

Huawei · AIShell Inc · Shanghai Jiao Tong University+4

Audio-Language Model Evaluation ASR Evaluation Spoken Language Understanding Speaker-Attributed ASR Automatic Speech Recognition

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have created new opportunities for wearable speech interfaces, with smart glasses providing an egocentric platform for continuous audio sensing and assistance. However, speech recognition and understanding in this setting remain challenging because of dynamic acoustic conditions, speaker overlap, and the spatial ambiguity introduced by wearer-centered recording geometry. To support systematic evaluation in this setting, we introduce the IEEE SLT 2026 SmartGlasses Challenge for egocentric multi-speaker speech processing. The challenge consists of two tracks, Dyadic Dialogue Understanding and Multi-party Meeting Understanding, and jointly evaluates Time-Stamped Speaker-Attributed Automatic Speech Recognition (TSA-ASR) and Spoken Language Understanding (SLU). It is built on a 106-hour four-channel egocentric speech dataset containing 714 sessions collected in real-world scenarios. This paper describes challenge tasks, dataset construction, submissions, and summarizes the main findings from the shared evaluation. The results show that heavy speaker overlap remains a major factor affecting TSA-ASR performance, while paralinguistic acoustic understanding continues to be difficult for current audio-language models in complex SLU settings. Further details can be found on the official challenge website.

Abstract

Speaker-Attributed ASR

Explore similar work

The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models

Phonology-Aware Speaker Extraction Frontend for Audio LLMs

G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition