We study whether persistent conversational speaker structure can be extracted directly from local overlapping speech mixtures. We propose a teacher-student framework that learns mixture-derived multi-speaker embeddings using only short overlapping segments and permutation-invariant latent supervision. Despite never being explicitly trained for speaker tracking, diarization, or conversational memory, the learned embedding space supports long-form speaker re-identification when combined with a lightweight online memory mechanism during inference. We additionally observe that the learned representation retains meaningful speaker structure under unseen overlap cardinalities. We further show that embeddings extracted from separation-first pipelines exhibit degraded clustering structure compared to embeddings predicted directly from mixtures. Finally, the learned embeddings remain effective for the downstream target speaker extraction task across multiple architectures. These findings suggest that local mixture-derived representations support persistent conversational speaker re-identification when combined with lightweight inference-time memory consolidation.
Figures & tables
τm
H
Confirm Rate ↑
Re-ID ↑
0.4
1
0.80
0.58
2
0.97
0.59
4
0.99
0.64
0.5
1
0.35
0.62
2
0.61
0.59
4
0.77
0.55
TABLE I : Sensitivity of online memory consolidation to threshold τm and reappearance window H , averaged over 150 simulated meetings.
Fig. 1 : System architecture and learned embedding structure.
Model
Params
Acc ↑
Sep ↑
NMI ↑
Sil ↑
ARI ↑
Single Speaker Case
ECAPA-TDNN [ 29 ]
15.9M
97.00
0.45
0.98
0.22
0.94
WavLM (Frozen) + EE
0.69M
96.49
0.46
0.98
0.22
0.93
WavLM (FT) + EE
47.9M
93.59
0.51
0.96
0.27
0.89
Dual Speaker Case
WavLM (Frozen) + K-means
–
7.35
0.008
0.05
-0.07
0.005
TABLE II : Clustering performance for the 2-speaker embedding model on Libri2Mix test set with modifications described in Section III-A . EE denotes the embedding extractor module (Fig. 1 ).
Train
Test
Acc ↑
Sep ↑
NMI ↑
Sil ↑
ARI ↑
2EE
2-spk
75.52
0.24
0.72
0.04
0.58
2EE
3-spk
64.30
0.21
0.60
0.03
0.44
3EE
2-spk
66.95
0.23
0.66
0.04
0.50
3EE
3-spk
60.77
0.20
0.58
0.03
0.41
TABLE III : Generalization across overlap cardinalities.
Method
Acc ↑
Sep ↑
NMI ↑
ARI ↑
Sil ↑
ConvTasNet + ECAPA-TDNN
11.00
0.014
0.076
0.011
-0.02
WavLM (FT) + 2EE
75.52
0.24
0.72
0.58
0.04
TABLE IV : Comparison against separation-first pipelines on Libri2Mix test set with modifications described in Section III-A .
Configuration
Confirm
Est./True
Re-ID ↑
Switch ↓
ARI ↑
Rate
Spk
Rate
Direct Online Assignment
1.00
33.2/10
0.50
0.70
0.32
Single-Speaker Guided Updates
0.45
7.90/10
0.75
0.37
0.43
Conservative Memory Consolidation
– Frozen WavLM + K-means
0.89
3.40/10
0.67
0.52
0.004
– ConvTasNet + ECAPA
0.98
2.00/10
0.66
0.47
0.01
TABLE V : Conversational memory behavior under different memory policies and embedding representations.
Fig. 2 : Conversational memory assignments across a representative long-form meeting. Each row corresponds to a ground-truth speaker. Colors indicate assigned memory identities, while hatched segments denote overlapping speech regions.
Method
Re-ID ↑
Switch ↓
NMI ↑
ARI ↑
Est./True
DER (%) ↓
Rate
Spk
Frozen WavLM + K-means
0.64
0.44
0.14
0.05
3.17 / 5
56.50
ConvTasNet + ECAPA
0.83
0.23
0.10
0.07
1.71 / 5
45.30
ECAPA(Intermediate Features)
+ 2EE
0.76
0.18
0.71
0.66
4.98 / 5
55.60
WavLM (FT) + 2EE
0.79
0.27
0.47
0.36
7.00 / 5
38.90
TABLE VI : Zero-shot tracking and diarization performance on the VoxConverse test set (English subset) without domain adaptation. All methods use the same online memory consolidation policy.
Model
Params
STOI ↑
PESQ ↑
SIG ↑
BAK ↑
OVRL ↑
SI-SDRi ↑
Dual Speaker Case
Noisy
-
0.74
1.18
2.98
2.69
2.33
-
pDCCRN [ 30 ]
– no embedding
3.7M
0.75
1.19
3.18
2.32
2.18
1.0
– concat
3.7M
0.78
1.37
2.62
3.21
2.16
5.8
– FiLM
3.8M
0.82
1.54
3.04
3.74
2.67
9.6
TABLE VII : Comparison of different TSE model architectures on Libri2Mix test set with modifications described in Section III-A .