cs.CLNov 14, 2026

Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards

Authors: Jing Pan, Ruizhi Li, Ju Lin, Jing Zheng, Florian Metze

Organizations: Meta, USA

Abstract

LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from 20.51%20.51\% for SFT to 13.14%13.14\%. Adding the delay objective reduces onset p50p50 by 65%65\% (from 0.918 s0.918\,\text{s} to 0.320 s0.320\,\text{s}) and p90p90 by 44%44\% (from 1.389 s1.389\,\text{s} to 0.780 s0.780\,\text{s}) relative to SA-WER-only RL, while achieving 12.70%12.70\% SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly.

Figures & tables

Explore similar work

CardsList
  1. TRADE: Transducer-Augmented Decoder for Speech LLM

    Jun 7, 2026Yun Tang, Shanil Puri, Shinji Watanabe +1Streaming ASRSpeech Processing

  2. Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognition

    Mar 11, 2026Yinfeng Xia, Jian Tang, Junfeng Hou +2Automatic Speech RecognitionStreaming ASR