Transcribing and understanding multi-speaker conversations requires speech recognition, speaker attribution, and timestamp localization. While speech LLMs excel at single-speaker tasks, multi-speaker scenarios remain challenging due to overlapping speech, backchannels, rapid turn-taking, and context window constraints. We propose Speaker-Reasoner, an end-to-end Speech LLM with agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window. Trained with a three-stage progressive strategy, Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets, particularly in handling overlapping speech and complex turn-taking.
Figures & tables
Figure 1: The overview of Speaker-Reasoner. The model employs an agentic multi-turn reasoning mechanism on the temporal axis, utilizing an indexing and slicing tool and a speaker-aware context cache to iteratively generate speaker identity, gender, timestamps, and transcription from raw multi-speaker audio.
Algorithm 1 Speaker-aware Cache Selection
Model
AISHELL4-Eval
Alimeeting-Far
DER ↓
CER ↓
cpCER ↓
Δ cp ↓
DER ↓
CER ↓
cpCER ↓
Δ cp ↓
Cascade Baselines
Pyannote3.1 + Paraformer
8.10
19.18
26.24
7.06
19.13
30.15
45.39
15.24
End-to-End Baselines
Gemini-2.5-Pro †
36.07
19.81
25.11
5.30
56.39
30.16
39.29
9.13
Qwen3-Omni-30B-A3B-Instruct
32.42
14.46
22.22
7.76
37.15
25.40
36.28
10.88
Table 1: Comparison of speaker diarization and recognition performance on AISHELL4-Eval and Alimeeting-Far test sets. The proposed models also serve as a stage-wise ablation across training stages. Bold denotes the best result in each column.
Model
AISHELL4-Eval
DER
cpCER
Gemini-2.5-Pro
15.32
31.59
Speaker-Reasoner Multi-turn w/ SAC
21.60
36.20
Table 2: Performance on long-form AISHELL4-Eval without segmentation.
Model
ACC
SCA
Gemini-2.5-Pro
94.80
67.03
Qwen3-Omni-30B-A3B-Instruct
97.12
60.49
Speaker-Reasoner Multi-turn
96.80
69.03
Table 3: Gender prediction accuracy (ACC) and speaker count accuracy (SCA) on AISHELL4-Eval. SCA measures whether the predicted number of speakers matches the ground truth.
We study timestamped speaker-attributed automatic speech recognition (SA-ASR) for long-form, multi-party speech with overlap. In this setting, chunk-wise inference must preserve meeting-level speaker identity consistency while producing time-stamped, speaker-labeled transcripts. Prior Speech-LLM systems tend to prioritize either local diarization or global labeling, lacking the ability to jointly model fine-grained temporal boundaries and robust cross-chunk identity linking. We propose G-STAR, an end-to-end framework that couples a cache-conditioned speaker-tracking module with a Speech-LLM transcription backbone. The tracker provides structured speaker cues with temporal grounding, and the LLM generates attributed text conditioned on these cues. G-STAR supports component-wise optimization and joint end-to-end training, enabling flexible learning under heterogeneous supervision and domain shift. Under chunk-wise decoding protocols, experiments on both oracle-segmented local evaluation and full-meeting global evaluation show strong speaker-attributed transcription performance.
Jing Peng, Ziyi Chen, Haoyu Li +7
Shanghai Jiao Tong University · Shenzhen Research Institute of Big Data · Nanjing University +2
Natural conversations make both speech recognition and speaker attribution challenging for ASR, as speakers take turns, overlap, and reappear over time. We propose STAM-ASR, Speaker-Temporal Anchoring with Memory, a lightweight framework that extends an already pretrained AudioLLM for multi-speaker ASR. Without relying on an external diarization system, STAM-ASR learns speaker activity and speaker-aware representations directly from intermediate AudioLLM features. Hence providing explicit who and when cues to modulate the AudioLLM's semantic representation without explicit speech separation. STAM-ASR further maintains fixed-size speaker and conversational memories to carry complementary context across turns. We evaluate STAM-ASR on AMI, ICSI, LibriCSS, and NOTSOFAR-1 across close-talk, far-field, overlapping, and cross-domain conditions. Our reported results shows that speaker-temporal conditioning and memory provide complementary benefits, while the gap between reference and predicted speaker activity identifies robust speaker tracking as a key remaining challenge.
Victor Tolulope Olufemi, Syeda Faiza Ahmed Sara, Shammur Absar Chowdhury
Qatar Computing Research Institute (QCRI), Doha, Qatar
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Dongwook Lee, Sangkwon Park, Eunwoo Song +6
IPAI, Seoul National University (SNU) · Department of ECE, SNU · Department of EE, Yonsei University +3