eess.ASJan 7, 2024
SaveDiarizationLM: Speaker Diarization Post-Processing with Large Language Models
Organizations: Google LLC, USA
Abstract
In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.
Figures & tables
Figure 1: The orchestration module associates each word from the ASR transcript with a speaker label from the speaker diarization outputs. (a) In this example, all words are associated with the correct speaker labels ( green arrows). The words “good”, “morning”, and “are” and “you” are associated with the only speaker label that overlap with them. The word “how” overlaps with both spk1 and spk2, but has bigger overlaps with spk2, thus is associated with spk2. The word “you” does not overlap with any speaker, but is closest to spk2, thus is associated with spk2. (b) In this example, two words are associated with wrong speaker labels ( red arrows) due to inconsistent timing information from the two systems. The word “how” is mistakenly associated with spk1, since spk1 has more overlap with this word than spk2. The word “you” is mistakenly associated with spk1, since spk1 is closer to this word than spk2.
Figure 2: Diagram of the proposed DiarizationLM framework.
Figure 3: The histogram for the number of hypothesis speakers predicted by the turn-to-diarize system on the Fisher training set. Note that the ground truth number of speakers is always two on the Fisher dataset, but we do not constrain the number of speakers for the turn-to-diarize system.
| System | Fisher testing set | Callhome testing set | ||||
| WER | WDER | cpWER | WER | WDER | cpWER | |
| USM + turn-to-diarize baseline | 15.48 | 5.32 | 21.19 | 15.36 | 7.72 | 24.39 |
| + PaLM 2-S zero-shot | - | 11.96 | 30.19 | - | 12.26 | 30.60 |
| + PaLM 2-S one-shot | - | 16.58 | 38.03 | - | 14.50 | 34.32 |
| + PaLM 2-L zero-shot | - | 11.36 | 31.78 | - | 13.29 | 34.30 |
| + PaLM 2-L one-shot | - | 5.94 | 22.21 | - | 7.95 | 24.67 |
Table 1: Evaluation results of the USM + turn-to-diarize baseline system and the results post-processed by DiarizationLM. For DiarizationLM, we experiment with PaLM 2 foundation models with and without finetuning on the diarization task. We also experiment with Llama 2 and Llama 3 models finetuned on the diarization task (model details in Appendix A ). WERs are the same for all systems due to TPST. All numbers are percentages.
| System | Fisher testing set | Callhome testing set | ||
|---|---|---|---|---|
| WDER | cpWER | WDER | cpWER | |
| Reference + turn-to-diarize baseline | 2.81 | 5.19 | 3.74 | 6.82 |
| + PaLM 2-S zero-shot | 7.50 | 12.70 | 7.29 | 12.79 |
| + PaLM 2-S one-shot | 10.92 | 19.16 | 12.79 | 21.65 |
| + PaLM 2-L zero-shot | 8.69 | 16.85 | 11.67 | 22.87 |
| + PaLM 2-L one-shot | 3.23 | 5.99 | 3.76 | 6.95 |
Table 2: Evaluation results of the turn-to-diarize baseline system with reference ASR transcript (assuming WER=0%) and the results post-processed by DiarizationLM. For DiarizationLM, we experiment with PaLM 2 foundation models with and without finetuning on the diarization task. All numbers are percentages.
| Utterance | Before DiarizationLM | After DiarizationLM |
|---|---|---|
| fe_03_07146 ( WDER =8.80%) | … spk:3 it’s it’s spk:1 more of summer always like you know we never experience a bit cold over here spk:4 usually it’ll spk:1 be warm or like very hot in summer yeah and spk:3 extremely hot yeah with high humidity my humidity is pretty spk:1 much high because i stay close to the sea coast over here spk:3 yeah spk:1 so spk:3 that makes you live houston is it like houston where you live yeah i i i live spk:1 in houston … | … spk:1 it’s it’s more of summer always like you know we never experience a bit cold over here usually it’ll be warm or like very hot in summer spk:2 yeah and extremely hot yeah with high humidity my spk:1 humidity is pretty much high because i stay close to the sea coast over here spk:2 yeah so that makes you live houston is it like houston where you live spk:1 yeah i i i live in houston … |
| fe_03_06816 ( WDER =6.61%) | … spk:3 uhuh spk:2 did you see the the woman golfer that was on this the one spk:1 monica yeah yeah spk:2 what’s her name monica stone yeah mhm she she spk:1 blew out she fell out of that tournament but i didn’t think she’d do it she she’s girls can’t compete against guys … | … spk:2 uhuh did you see the the woman golfer that was on this the one spk:1 monica yeah yeah spk:2 what’s her name monica stone spk:1 yeah spk:2 mhm spk:1 she she blew out she fell out of that tournament but i didn’t think she’d do it she she’s girls can’t compete against guys … |
| fe_03_11210 ( WDER =6.35%) | … spk:1 the vikings mine’s the eagles i’m from new jersey oh all right i have my jersey on now i watch the game tonight yeah well i i may i may just watch spk:2 part of it tonight too then but uh it’s a case as i say if if i had to pay for it i probably wouldn’t watch it spk:1 i wouldn’t either uhhuh spk:2 unless spk:1 it was an eagles game … | … spk:1 the vikings mine’s the eagles i’m from new jersey spk:2 oh all right spk:1 i have my jersey on now i watch the game tonight yeah spk:2 well i i may i may just watch part of it tonight too then but uh it’s a case as i say if if i had to pay for it i probably wouldn’t watch it spk:1 i wouldn’t either spk:2 uhhuh spk:1 unless it was an eagles game … |
| fe_03_11159 ( WDER =4.05%) | … spk:2 yeah spk:1 anniversary that’s horrible spk:2 yeah spk:1 yeah it’s not good spk:2 i spk:1 i hear i hear you there that’s not a good thing you spk:2 know i mean of course you know that’s a day that will go down instantly nobody will ever remember it … | … spk:1 yeah anniversary that’s horrible yeah yeah it’s not good i i hear i hear you there that’s not a good thing spk:2 you know i mean of course you know that’s a day that will go down instantly nobody will ever remember it … |
Table 3: Example cases from the Fisher testing set where we see big absolute WDER reduction ( WDER) with DiarizationLM (deg2ref flavor).
| Utterance | Before DiarizationLM | After DiarizationLM |
|---|---|---|
| en_6447 ( WDER =12.49%) | … spk:1 i’m spk:2 going to see if i can talk to the guy that’s selling the trailer if i can chew him down a bit uhhuh spk:1 and spk:2 you know what you just said benedicta is are you living with benedicta spk:1 yes yes yes spk:2 you know what i bet she answered the phone … | … spk:2 i’m going to see if i can talk to the guy that’s selling the trailer if i can chew him down a bit spk:1 uhhuh spk:2 and you know what you just said benedicta is are you living with benedicta spk:1 yes yes yes spk:2 you know what i bet she answered the phone … |
| en_6408 ( WDER =10.87%) | … spk:1 uhu spk:2 so spk:1 he had big surgery again and he’s in a wheelchair oh my spk:2 and spk:1 he doesn’t want to go to school in a wheelchair uhuh but spk:2 he might he wants to have tutoring at home but they’re still where they lived on 45th street spk:1 yeah they’re there … | … spk:2 uhu spk:1 so he had big surgery again and he’s in a wheelchair spk:2 oh my spk:1 and he doesn’t want to go to school in a wheelchair spk:2 uhuh spk:1 but he might he wants to have tutoring at home spk:2 but they’re still where they lived on 45th street spk:1 yeah they’re there … |
| en_6298 ( WDER =9.95%) | … spk:1 um hey we’re we’re confused about you guys address spk:2 is spk:1 it 1324 or 13 spk:2 it’s 1 324 spk:1 excuse me 1324 yes and it’s me view is me two words or one word yes it’s two words and there’s an ln besides … | … spk:1 um hey we’re we’re confused about you guys address is it 1324 or 13 spk:2 it’s 1 324 spk:1 excuse me 1324 spk:2 yes spk:1 and it’s me view is me two words or one word spk:2 yes it’s two words and there’s an ln besides … |
| en_4792 ( WDER =9.42%) | … spk:2 yeah well he was at columbia spk:1 he was there like five years and they turned him down for tenure then he went somewhere else he he was down in college park maryland yeah and he i think he was only non tenure track down there then supposedly supposed to be back in japan now yeah but you know he’s he’s probably become an english teacher at some unit yeah i know a guy believe it or not i know a guy from manhattan who was up in sapotto his major he did an mba believe it or not he’s he’s an english teacher now huh … | … spk:2 yeah well he was at columbia spk:1 he was there like five years and they turned him down for tenure then he went somewhere else he he was down in college park maryland spk:2 yeah spk:1 and he i think he was only non tenure track down there then supposedly supposed to be back in japan now spk:2 yeah spk:1 but you know he’s he’s probably become an english teacher at some unit spk:2 yeah spk:1 i know a guy believe it or not i know a guy from manhattan who was up in sapotto his major he did an mba believe it or not he’s he’s an english teacher now spk:2 huh … |
Table 4: Example cases from the Callhome testing set where we see big absolute WDER reduction ( WDER) with DiarizationLM (deg2ref flavor).
| Utterance | Before DiarizationLM | After DiarizationLM |
|---|---|---|
| fe_03_11252 | … spk:1 oh okay i believe it’s a lot wrong with the public schools i don’t believe that they’re um that they’re giving um these kids a sense of um well they’re not teaching them what they need to know once they get out of um school you know mhm um what what’s happening is that’s probably why you got a lot of um a lot of people that’s unemployed i think you know they you get a lot from school and they taking a lot of um i guess the economics out of school you know spk:2 right … | … spk:1 oh okay i believe it’s a lot wrong with the public schools i don’t believe that they’re um that they’re giving um these kids a sense of um well they’re not teaching them what they need to know once they get out of um school you know spk:2 right … |
| fe_03_11224 | … spk:1 so um i think what do you think is an important thing in a relation i think the topic was um what you um what are the most important things in a life partner yeah uh h well what do you think me spk:2 i would have to say trust and honesty like cuz without that you really don’t have nothing to build on you know right yeah … | … spk:1 so um i think what do you think is an important thing in a relation spk:2 i would have to say trust and honesty like cuz without that you really don’t have nothing to build on you know right … |
Table 5: Example cases from the Fisher testing set where zero-shot PaLM 2-S deletes lots of text from the prompt.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Modification Span Length | Words Fixed | Words Broken | Net Word Gain | Linguistic Regime |
| word | Single-word turn boundaries / interjections | |||
| words | Two-word backchannels ( “mm-hmm” , “oh yeah” ) | |||
| words | Short 3-word turn handoffs | |||
| words | Short 4-word boundary clauses | |||
| words | Short 5-word boundary clauses | |||
| Short Spans ( words) | 585 | 397 | +188 | Lexically disambiguated boundary fixes |
Table 6: Word-level forensic decomposition of LLM speaker modifications by contiguous span length on the AMI test set (16 meetings, 78,943 words) for Gemma 4 E4B trained with Standard SFT.
| Fisher (172) | Callhome (20) | ICSI (3) | AMI (16) | ||||||
| System / Training Method | Post-Proc. | WDER | cpWER | WDER | cpWER | WDER | cpWER | WDER | cpWER |
| USM + turn-to-diarize (Baseline) | — | 5.32 | 20.88 | 7.74 | 24.31 | 14.70 | 43.90 | 15.68 | 40.32 |
| DiarizationLM-8b-Fisher-v2 (8B) | Raw TPST | 3.28 | 18.37 | 6.66 | 23.57 | — | — | — | — |
| Gemma 4 E4B (Standard SFT, step 25k) | Raw TPST | 2.95 | 17.49 | 6.22 | 21.94 | 14.87 | 44.68 | 17.61 | 43.93 |
| Gemma 4 E4B (Standard SFT, step 25k) | LG-TPST | 3.06 | 17.64 | 5.92 | 21.63 | 13.85 | 43.23 | 14.82 | 39.21 |
| Gemma 4 E4B (SFT-Anchored DPO, step 500) | Raw TPST | — | — | 9.88 | 26.10 | 16.37 | 45.09 | 16.76 | 40.28 |
Table 7: Ablation of training algorithms (Standard SFT, SFT-Anchored DPO, and Locality-Preserving SFT) and inference post-processing (Raw TPST vs. LG-TPST) on Gemma 4 E4B across all four benchmarks.
| Dataset | Metric | Baseline | Gemma 4 E4B (4B Dense) | Gemma 3 12B (12B Dense) | Gemma 4 26B_A4B (26B MoE) |
|---|---|---|---|---|---|
| Fisher | WDER (%) | 5.32 [4.93, 5.74] | 2.99 [2.65, 3.37] | 2.78 [2.45, 3.16] | 2.89 [2.56, 3.26] |
| cpWER (%) | 20.88 [19.97, 21.88] | 17.62 [16.76, 18.56] | 17.32 [16.46, 18.25] | 17.46 [16.62, 18.37] | |
| SpkCntMAE | 0.215 [0.151, 0.291] | 0.093 [0.047, 0.145] | 0.093 [0.047, 0.145] | 0.105 [0.058, 0.157] | |
| Callhome | WDER (%) | 7.74 [6.07, 9.65] | 4.92 [3.46, 6.75] | 5.00 [3.38, 6.89] | 5.52 [3.71, 7.71] |
| cpWER (%) | 24.31 [21.27, 27.43] | 20.69 [17.98, 23.66] | 20.73 [17.89, 23.80] | 21.45 [18.33, 24.84] | |
| SpkCntMAE | 0.050 [0.000, 0.150] | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] |
Table 8: Comparison across model sizes and architectures on Fisher, Callhome, ICSI, and AMI, reporting point estimates and 95% non-parametric bootstrap confidence intervals ( resamples).
| Paired Comparison ( ) | Dataset | WDER (%) [95% CI] | -value | cpWER (%) [95% CI] | -value |
| Gemma 4 E4B (4B) vs. Baseline | Fisher | -2.33 [-2.51, -2.16] | -3.26 [-3.52, -3.01] | ||
| Callhome | -2.82 [-3.48, -2.14] | -3.62 [-4.39, -2.84] | |||
| ICSI | -0.60 [-0.88, -0.35] | -0.58 [-0.72, -0.47] | |||
| AMI | -0.79 [-1.00, -0.60] | -0.75 [-0.94, -0.57] | |||
| Gemma 3 12B (12B) vs. Gemma 4 E4B (4B) | Fisher | ||||
| Callhome |
Table 9: Paired bootstrap differences ( , 95% CI, and two-sided -value over resamples) evaluating (1) Gemma 4 E4B (Locality-SFT) vs. Baseline, and (2) larger models (12B Dense and 26B MoE) vs. Gemma 4 E4B.
More in this topic
Speaker Diarization
Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.
See all 55 papers