Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards
Organizations: Meta, USA
Abstract
LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from for SFT to . Adding the delay objective reduces onset by (from to ) and by (from to ) relative to SA-WER-only RL, while achieving SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly.
Figures & tables
| System | SA-WER | ||||
|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | |
| SFT | 20.51 | 3.2 | 5.0 | 12.2 | 14.4 |
| SFT oracle@8 | 14.61 | – | – | – | – |
| SA-WER RL | 13.14 | 0.5 | 0.3 | 9.5 | 8.1 |
| Buffer | SA-WER | ||||
|---|---|---|---|---|---|
| (ms) | (%) | (%) | (%) | (%) | (%) |
| None | 13.14 | 0.5 | 0.3 | 9.5 | 8.1 |
| 1,300 | 13.41 | 0.3 | 1.1 | 9.6 | 8.2 |
| 1,000 | 15.51 | 1.1 | 4.8 | 9.7 | 8.1 |
| 700 | 13.36 | 0.5 | 1.3 | 9.2 | 7.8 |
| 400 | 12.70 | 0.3 | 1.0 | 8.8 | 7.4 |
| SA-WER | Neg. | ||||
|---|---|---|---|---|---|
| (%) | (s) | (s) | (s) | (%) | |
| 0.05 | 13.17 | 0.850 | 1.336 | 1.427 | 2.46 |
| 0.20 | 13.36 | 0.680 | 1.149 | 1.240 | 2.50 |
Streaming ASR
Streaming automatic speech recognition (ASR) transcribes speech as it arrives while balancing accuracy, latency, and computing cost.
See all 48 papers