CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
Authors: Liangbin Huang, Xiaohua Liao, Chaoqun Cui, Shijing Wang, Zhaolong Huang, Yanlong Du, Wenji Mao
Organizations: Hujing Digital Media and Entertainment Group · MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Computer Science and Technology, Beijing Jiaotong University
Traditional speaker diarization systems have primarily focused on constrained scenarios such as meetings and interviews, where the number of speakers is limited and acoustic conditions are relatively clean. To explore open-world speaker diarization, we extend this task to the visual media domain, encompassing complex audiovisual programs such as films and TV series. This new setting introduces several challenges, including long-form video understanding, a large number of speakers, cross-modal asynchrony between audio and visual cues, and uncontrolled in-the-wild variability. To address these challenges, we propose Cinematic Speaker Registration & Diarization (CineSRD), a unified multimodal framework that leverages visual, acoustic, and linguistic cues from video, speech, and subtitles for speaker annotation. CineSRD first performs visual anchor clustering to register initial speakers and then integrates an audio language model for speaker turn detection, refining annotations and supplementing unregistered off-screen speakers. Furthermore, we construct and release a dedicated speaker diarization benchmark for visual media that includes Chinese and English programs. Experimental results demonstrate that CineSRD achieves superior performance on the proposed benchmark and competitive results on conventional datasets, validating its robustness and generalizability in open-world visual media settings.
Figures & tables
Figure 1: The overall structure of the unified framework CineSRD for visual media speaker diarization.
Figure 2: Demonstration of speaker registration.
Group
Line
Similarity
Active
σ(s)
σ(G)
Operation
1
You’ve got a good heart.
-
✓
1.0
1.0
-
Don’t ever lose it.
0.651
✓
1.0
No matter what happens.
0.712
✓
1.0
2
Power, real power,
0.198
✗
0.185
0.210
Register new speaker
doesn’t come to those who were born strongest.
0.684
✗
0.235
3
I thought maybe you could love me like you used to.
0.155
✓
1.0
1.0
-
Table 1: The new speaker supplementation process. Similarity represents the timbre embedding similarity of adjacent lines.
Statistic
Chinese
Chinese-Hard
English
language
CN
CN (dialect)
EN
# duration (h)
70.98
16.96
4.65
# avg speaker
127.83
317
19.5
# lines
80752
15528
5227
# avg line token
6.14
6.92
9.05
Table 2: Statistics of SubtitleSD dataset.
Dataset
Modality
Scenario
# Videos
Total Duration
ASV
TDS
AMI Corpus [ 4 ]
AV
meetings
684
100h
4.0
189
AVDIAR [ 18 ]
AV
chat
27
21m
2.2
11
VoxConverse [ 10 ]
A
debate, news
448
63h50m
5.6
-
AVA-AVD [ 34 ]
AV
documentaries, moviesbie
351
29h15m
7.7
1500
SubtitleSD (Ours)
AVT
multi-genre visual media
130
92h35m
21.2
1054
Table 3: Comparisons with existing speaker diarization datasets/benchmarks. ASV denotes the average number of speakers per video. TDS denotes the total number of distinct speakers.
Type
Setting
Value
Remark
Models
active speaker detection
TalkNet [ 29 ]
-
face detection
RetinaFace [ 14 ]
-
face embedding
CurricularFace [ 20 ]
-
face quality assessment
FQA
from Alibaba DAMO Academy in ModelScope
timbre embedding
ERes2NetV2 [ 5 ]
-
ALM
Qwen2-Audio-7B [ 9 ]
the ALM used in speaker turn detection
Table 4: Hyperparameter configuration in experiments.
Chinese
Chinese-Hard
English
Method
Modality
DER ↓
JER ↓
DER ↓
JER ↓
DER ↓
JER ↓
AHC [ 3 ]
A
0.13982
0.45223
0.20679
0.57714
0.12481
0.41017
SC [ 31 ]
A
0.14945
0.47114
0.22255
0.65194
0.20116
0.43402
EC2P [ 6 ]
AV
0.13595
0.41524
0.21125
0.58984
0.11951
0.39785
AVT
0.13451
0.38012
0.21019
0.55684
0.11804
0.35571
CineSRD (Ours)
AV
0.08325
0.41444
0.10180
0.55392
0.10272
0.31331
Table 5: Speaker diarization evaluation in open-world visual media scenario. CineSRD adopts Spectral Clustering to perform audio-visual modality clustering. The 1st and 2nd best results are denoted as blue and orange , respectively.
Method
Modality
DER ↓
SPKE ↓
AHC [ 3 ]
A
0.2137
0.1845
SC [ 31 ]
A
0.2131
0.1839
AVR-Net [ 34 ]
AV
0.2057
0.1765
EC2P [ 6 ]
AV
0.2032
0.1740
CineSRD (AHC)
AV
0.1951
0.1630
CineSRD (SC)
AV
0.1898
0.1616
Table 6: Evaluation results of AVA-AVD datasets. The performance metrics of the baseline methods are obtained from their respective papers.
Chinese
Chinese-Hard
English
Clustering
Modality
DER ↓
JER ↓
DER ↓
JER ↓
DER ↓
JER ↓
AHC
A
0.13982
0.45223
0.20679
0.57714
0.12481
0.41017
AV
0.08528
0.41847
0.10585
0.55233
0.10554
0.31771
AVT
0.07700
0.32119
0.09928
0.41314
0.10357
0.27094
SC
A
0.14945
0.47114
0.22255
0.65194
0.20116
0.43402
AV
0.08325
0.41444
0.10180
0.55392
0.10272
0.31331
Table 7: CineSRD performance across modalities using different clustering methods. The setting that uses only the textual modality refers to the pure clustering method.
Method
Modality
AUC ↑
F1 ↑
ALM Only
AT
0.78974
0.83015
Audio Only
A
0.95862
0.86490
ALM + Audio
AT
0.96257
0.92715
Table 8: Evaluation results of different methods on speaker turn detection.
Figure 3: Label frequency distribution on Chinese subset.
Figure 4: Impact of weight w in speaker turn detection.
Figure 5: Impact of Threshold η in off-screen speaker supplementation.
While Omni-modal Large Language Models have made strides in joint sensory processing, they fundamentally struggle with a cornerstone of human interaction: deciphering complex, multi-person conversational dynamics to accurately answer Who said what and when.'' Current models suffer from an illusion of competence'' -- they exploit visual biases in conventional benchmarks to bypass genuine cross-modal alignment, while relying on sparse, low-frame-rate visual sampling that destroys crucial high-frequency dynamics like lip movements. To shatter this illusion, we introduce Visual-Registered Speaker Diarization and Recognition (VR-SDR) and the HumanOmni-Speaker Benchmark. By strictly eliminating visual shortcuts, this rigorous paradigm demands true end-to-end spatio-temporal identity binding using only natural language queries. To overcome the underlying architectural perception gap, we propose HumanOmni-Speaker, powered by a Visual Delta Encoder. By sampling raw video at 25 fps and explicitly compressing inter-frame motion residuals into just 6 tokens per frame, it captures fine-grained visemes and speaker trajectories without triggering a catastrophic token explosion. Ultimately, HumanOmni-Speaker demonstrates strong multimodal synergy, natively enabling end-to-end lip-reading and high-precision spatial localization without intrusive cropping, and achieving superior performance across a wide spectrum of speaker-centric tasks.
Detao Bai, Zhiheng Ma, Xihan Wei
Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1
Long-form TV dramas present a formidable challenge for comprehensive video understanding, where deciphering complex storyline often relies on speaker recognition, the task of accurately attributing each spoken utterance to its respective character. In this paper, we advance this field through two primary contributions. (1) We introduce DramaSR-532K, a large-scale benchmark comprising 532K annotated dialogue lines across more than 900 unique characters, necessitating the integration of auditory, linguistic, and visual cues for speaker recognition. (2) We propose DramaSR-LRM, a robust approach built upon a large reasoning model (LRM). DramaSR-LRM is designed to autonomously aggregate contextual evidence via multimodal tool-use, synthesizing diverse inputs to achieve high-fidelity attribution. Experimental results demonstrate that DramaSR-LRM significantly outperforms existing baselines, particularly on short utterances where acoustic biometrics are inherently unreliable. All the data and code will be made publicly available at the project page: https://www.github.com/198808xc/DramaSR-LRM.
Yuxuan Li, Lingxi Xie, Xinyue Huo +6
Tsinghua University, China · Huawei Inc., China · University of Chinese Academy of Sciences, China +1
Multi-speaker automatic speech recognition (MSASR) aims to jointly predict content transcriptions, speaker identities, and timestamps, thereby addressing the key question of "who spoke what and when" and holds substantial practical value in real-world multi-speaker scenarios. However, MSASR still encounters considerable challenges in the presence of fast turn transitions, overlapping speech, and complex, diverse multi-speaker scenarios. In this work, we propose DiaScriber, an end-to-end multi-speaker diarization and transcription model built on a speech large language model. We first construct diverse data pipelines to cover a wide variety of multi-speaker scenarios and their complexities, including validation and refinement, turn-transition and overlapping-speech simulation, and multimodal annotation. Furthermore, DiaScriber is developed based on the pretrained version of Qwen3.5-Omni through a three-stage training strategy involving continual pretraining, supervised fine-tuning, and reinforcement learning. Experiments show that DiaScriber achieves superior performance over comparison methods across extensive multi-speaker scenario test sets and demonstrates outstanding generalization ability in unseen multi-speaker scenarios.