Off-target posterior mass: A training-free diagnostic for fine-tuning spoken language diarization probes
Organizations: Signal and Image Processing Institute, Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, California 90089, USA · Language Technologies Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA
Abstract
Spoken Language Diarization (SLD) models provide frame-level language information of a speech recording. SLD modeling pipeline includes feature extraction from pre-trained speech models followed by clustering. Fine-tuning pre-trained speech models has shown performance gains for SLD with its magnitude being dataset dependent. Off-target Posterior Mass (OTPM), computed based on posterior language identification probabilities is proposed as a diagnostic metric to forecast performance gains from SLD fine-tuning. OTPM obtains positive correlation with probe gains, 0.56 on unseen synthetic bilingual data and 0.23 on real-world corpora. These results indicate promising potential of OTPM as a diagnostic metric for SLD.
Figures & tables
| Probe | ||||||
| Posterior, all | ||||||
| Posterior, non-coll. | ||||||
| Encoder, all |
| Predicted | Within-corpus | |||||||
| Corpus | OTPM | LDER | ( ) | ( ) | Encoder | Posterior | ||
| SEAME | 0.392 | 68.61 | (0.003) | (0.002) | ||||
| VoxPopuli | 0.374 | 25.82 | (0.004) | (0.003) | ||||
| DISPLACE | 0.458 | 37.05 | (0.17) | (0.0025) | ||||
| Average over corpora (one-sided ) | (0.0015) | (0.012) | ||||||
| Posterior probe | Encoder probe | ||||||||||
| [95% CI] | [95% CI] | ||||||||||
| base | 28 | 32 | |||||||||
| large-v3 | 31 | 32 | |||||||||
Speaker Diarization
Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.
See all 66 papers