Transcribing and understanding multi-speaker conversations requires speech recognition, speaker attribution, and timestamp localization. While speech LLMs excel at single-speaker tasks, multi-speaker scenarios remain challenging due to overlapping speech, backchannels, rapid turn-taking, and context window constraints. We propose Speaker-Reasoner, an end-to-end Speech LLM with agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window. Trained with a three-stage progressive strategy, Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets, particularly in handling overlapping speech and complex turn-taking.
Figures & tables
Figure 1: The overview of Speaker-Reasoner. The model employs an agentic multi-turn reasoning mechanism on the temporal axis, utilizing an indexing and slicing tool and a speaker-aware context cache to iteratively generate speaker identity, gender, timestamps, and transcription from raw multi-speaker audio.
Algorithm 1 Speaker-aware Cache Selection
Model
AISHELL4-Eval
Alimeeting-Far
DER ↓
CER ↓
cpCER ↓
Δ cp ↓
DER ↓
CER ↓
cpCER ↓
Δ cp ↓
Cascade Baselines
Pyannote3.1 + Paraformer
8.10
19.18
26.24
7.06
19.13
30.15
45.39
15.24
End-to-End Baselines
Gemini-2.5-Pro †
36.07
19.81
25.11
5.30
56.39
30.16
39.29
9.13
Qwen3-Omni-30B-A3B-Instruct
32.42
14.46
22.22
7.76
37.15
25.40
36.28
10.88
Table 1: Comparison of speaker diarization and recognition performance on AISHELL4-Eval and Alimeeting-Far test sets. The proposed models also serve as a stage-wise ablation across training stages. Bold denotes the best result in each column.
Model
AISHELL4-Eval
DER
cpCER
Gemini-2.5-Pro
15.32
31.59
Speaker-Reasoner Multi-turn w/ SAC
21.60
36.20
Table 2: Performance on long-form AISHELL4-Eval without segmentation.
Model
ACC
SCA
Gemini-2.5-Pro
94.80
67.03
Qwen3-Omni-30B-A3B-Instruct
97.12
60.49
Speaker-Reasoner Multi-turn
96.80
69.03
Table 3: Gender prediction accuracy (ACC) and speaker count accuracy (SCA) on AISHELL4-Eval. SCA measures whether the predicted number of speakers matches the ground truth.