Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
Abstract
Full-duplex spoken dialogue models learn turn-taking, overlaps and backchannels from conversations recorded with one channel per speaker, but most available conversational audio is a single mono mix. We present Kezek, a model that turns a mono recording of exactly two people into one track per person with the original length and timing; overlaps and short replies ("yeah", "mm-hm") stay on the track of the person who said them. Kezek decides who speaks when and who is who once for the whole recording (stage 1: a two-voice activity model guided by a speaker fingerprint trained to be insensitive to phase), and a conditioned time-domain separator only carries out that decision (stage 2), with outputs that always sum to the input. Standard separation metrics miss the failures that matter for dialogue data, so we report short-reply retention, leakage onto the other person's solo, channel swaps and coverage. Against the open pipeline pyannote -> DialogueSidon on the same recordings, Kezek keeps more short replies (0.83 vs. 0.69 on Open Yap test; 0.80 vs. 0.71 over the other's speech on unseen telephone speech from CallHome English), mostly because it returns the whole recording while the external pipeline drops part of it (on the audio both return, reply retention is about the same: 0.859 vs. 0.865 on Open Yap); it also leaks less, swaps channels less often, and runs 4-5 times faster (about 230 hours of audio per GPU-hour). We also report negative results and an ideal-mask ceiling that locate the remaining gap: quiet replies under a louder voice. Code and weights are released under permissive licences.
Figures & tables
| Activity F1 | Short replies | No-swap conv. | Wrong track | ||||
|---|---|---|---|---|---|---|---|
| Labels | clean | rec | clean | rec | clean | rec | EN / KK |
| Energy VAD | 0.877 | 0.827 | 0.731 | 0.711 | 63.7% | 54.0% | 25.5% / 28.6% |
| Word-gated | 0.952 | 0.943 | 0.896 | 0.862 | 97.3% | 96.5% | 3.6% / 0% |
| Reference activity | 0.993 | 0.992 | — | — | 100% | 100% | — |
| Stage | Params | Steps | Synthetic | Window | Pipeline |
|---|---|---|---|---|---|
| TF-GridNet S, Open Yap, scratch | 2.0 M | 2.5 k | — | +0.37 / | — |
| M, Libri-Light overlaps, scratch | 13.3 M | 50 k | +9.68 | +5.28 / +3.49 | — |
| M, Open Yap mixture | 13.3 M | 17.5 k | +9.98 | +7.55 / +5.09 | +9.49 / +7.01 |
| L, Open Yap mixture, scratch | 19.6 M | 12.5 k | +4.74 | +2.73 / +1.86 | — |
| L, widened from M | 19.6 M | 27.5 k | +10.75 | +8.75 / +6.15 | +9.86 / +7.65 |
| L, on our fingerprint profiles | 19.6 M | 15 k | — | +9.02 / +6.00 | +10.11 / +7.67 |
| Corpus | Used for | Terms |
|---|---|---|
| Open Yap 1K [ 18 ] | training, validation, test | Data Use Agreement v1: commercial use and model distribution allowed; no redistribution of audio |
| Libri-Light [ 6 ] | synthetic overlaps, fingerprint | LibriVox audiobooks (public domain); code MIT |
| VoxPopuli [ 19 ] | fingerprint | data CC0; European Parliament material, source acknowledged |
| MUSAN [ 17 ] | noise and music in the recording chain | CC BY 4.0; content selected to allow commercial use |
| RIRS_NOISES [ 7 ] | reverberation | Apache 2.0 |
| CallHome English [ 1 , 9 ] | evaluation only | CC BY-NC-SA 4.0 (TalkBank) |
| Replies kept | ||||||||
|---|---|---|---|---|---|---|---|---|
| Set | System | all | over speech | in pause | Leak (dB) | No-swap | Coverage | h / GPU-h |
| Open Yap test | External | 0.687 | — | — | 0.793 | 0.735 | 51 | |
| Kezek | 0.830 | — | — | 0.934 | 1.000 | 231 | ||
| CallHome | External | — | 0.711 | 0.798 | 0.797 | 0.816 | 46 | |
| Kezek | — | 0.796 | 0.907 | 0.859 | 1.000 | 232 | ||
| Clean | As recorded | |||
| Metric | Prototype | Version 1 | Prototype | Version 1 |
| Overlap SI-SDRi (dB) | 10.08 | 10.49 | 8.57 | 8.41 |
| Solo SI-SDR (dB) | 21.59 | 21.46 | 19.41 | 19.30 |
| Leak (dB) | ||||
| Short replies kept | 0.900 | 0.864 | 0.871 | 0.811 |
| Speed (h / GPU-h) | 118 / 231 | |||
| clean | full chain | reverb | noise | codec | telephone | |
| Activity F1 | ||||||
| Prototype | 0.938 | 0.928 | 0.925 | 0.920 | 0.939 | 0.937 |
| PairNet | 0.938 | 0.924 | 0.923 | 0.919 | 0.937 | 0.932 |
| Short replies | ||||||
| Prototype | 0.904 | 0.886 | 0.873 | 0.846 | 0.901 | 0.893 |
| PairNet | 0.886 | 0.873 | 0.853 | 0.860 | 0.884 | 0.873 |
| Set | System | DER | Miss | FA | Conf. | JER | 2 spk. | h / GPU-h |
|---|---|---|---|---|---|---|---|---|
| Open Yap, clean | pyannote | 25.11 (19.65) | 17.78 | 4.29 | 3.03 | 27.69 | 0.835 | 150 |
| Kezek stage 1 | 15.27 (10.87) | 3.04 | 11.71 | 0.51 | 15.84 | 1.000 | 2 400 | |
| Open Yap, as rec. | pyannote | 25.79 (20.51) | 16.14 | 5.21 | 4.44 | 29.30 | 0.851 | 154 |
| Kezek stage 1 | 19.31 (14.42) | 3.23 | 15.26 | 0.82 | 18.94 | 1.000 | 2 600 | |
| CallHome | pyannote | 18.44 (13.05) | 10.48 | 4.33 | 3.64 | 23.00 | 1.000 | 153 |
| Kezek stage 1 | 30.05 (20.42) | 5.06 | 23.00 | 1.99 | 27.94 | 1.000 | 2 300 |
| Clean | As recorded | |||||
|---|---|---|---|---|---|---|
| Pipeline | Overlap | Leak | Replies | Overlap | Leak | Replies |
| Prototype | +9.77 | 0.896 | +7.69 | 0.851 | ||
| Prototype stage 1 + our fingerprint and separator | +10.26 | 0.896 | +7.88 | 0.818 | ||
| Version 1 | +10.11 | 0.851 | +7.67 | 0.792 | ||
| Version 1 + short-burst rescue | +10.14 | 0.857 | +7.66 | 0.799 | ||
| System | All | Louder | 0 to dB | Below dB |
|---|---|---|---|---|
| Ideal ratio mask | 0.938 (0.944) | 1.000 (1.000) | 1.000 (1.000) | 0.743 (0.784) |
| Kezek version 1 | 0.722 (0.804) | 0.965 (0.968) | 0.827 (0.907) | 0.171 (0.405) |
| Separator + labelled activity | 0.757 (0.804) | 0.947 (1.000) | 0.865 (0.860) | 0.286 (0.405) |
| Mixture on both tracks | 0.396 (0.441) | 0.982 | 0.019 | 0.000 |
| Change | Result (vs. the configuration it was compared with) |
|---|---|
| TF-GridNet [ 20 ] (2 M) from scratch on natural windows | learns to route solo, overlap SI-SDRi +0.4 dB |
| Separator 19.6 M 26 M | +7.68 vs. +7.67 dB overlaps, leak unchanged, replies 0.805 vs. 0.792 |
| Separator without activity hints | replies over speech 0.662 vs. 0.708, leak vs. dB, swaps in 4 of 20 recordings vs. 0 |
| Separator fine-tuned on hints of the real stage 1, leak weight 3.3 | leak +0.35 dB better, overlaps dB, replies |
| Synthetic replies 5–25 dB below the main voice | quiet replies 0.397 vs. 0.371 (ideal mask 0.788; all 113 val conversations) |
| Short-reply loss weight 3 in the separator | replies 0.825 vs. 0.838, overlaps dB |
| Change | Open Yap | Libri | Libri hard | Hard | |
|---|---|---|---|---|---|
| ECAPA-TDNN | — | 11.7 | 6.9 | 11.2 | 15.0 |
| #1 | raw audio, siblings, channel-matched pairs | 15.3 | 7.7 | 10.8 | 15.2 |
| #2 | + full chain, 25% conversational anchors | 15.7 | 6.5 | 9.4 | 15.0 |
| #3 | + telephone band on 35% of crops | 14.9 | 6.3 | — | 14.7 |
| #4 | + telephone band of random shape | 14.6 | 5.9 | — | 14.3 |
| #5 | + random all-pass on 60% of crops | 13.6 | 5.2 | 7.2 | 13.2 |
| Filter on half of the crops | ECAPA | Ours, #4 | Ours, #5 (+ all-pass training) |
|---|---|---|---|
| telephone band, zero phase | 9.2 | 8.5 | — |
| telephone band, IIR (with phase) | 9.1 | 14.1 | 10.8 |
| all-pass only ( ) | 7.5 | 9.6 | 8.0 |
| telephone band, resampled to 8 kHz | 10.8 | 14.6 | 11.2 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Architecture | Params |
|---|---|---|
| Fingerprint | conv front end (strides 5-4-4-4) Conformer (6 layers, 256-d) attentive statistics pooling 192-d | 10.1 M |
| PairNet | fingerprint-style encoder + log-mel branch 128-d voice embedding per 20 ms 2 Transformer decoder layers | 11.8 M |
| KezekTSE-L | Conv-TasNet: , , stride 16, , , , , gLN, FiLM by both profiles in every block, hint via 1 1 conv at the bottleneck, two softmax masks | 19.6 M |
| Stage | LR | Warmup | Steps | Batch | Crop |
|---|---|---|---|---|---|
| Fingerprint (AAM-softmax, , ) | |||||
| first version | 1e-3 | 2 000 | 24.8 k | 128 pairs | 1–3 s |
| full chain, conversational anchors | 5e-4 | 500 | 12 k | 128 pairs | 1–3 s |
| telephone band 35% | 3e-4 | 300 | 6 k | 128 pairs | 1–3 s |
| random telephone band | 3e-4 | 300 | 6 k | 128 pairs | 1–3 s |
| random all-pass | 3e-4 | 300 | 10 k | 128 pairs | 1–3 s |
| Set | System | Replies | Leak (dB) | No-swap | Delay |
|---|---|---|---|---|---|
| Open Yap test | reference tracks as output | 1.000 | / (floor) | 1.000 | 0 ms |
| mixture on both tracks | 0.994 / 1.000 | / | — | — | |
| CallHome | mixture gated by the annotation | 1.000 / 1.000 | 1.000 | — | |
| mixture on both tracks | 0.167 / 1.000 | 0.0 | — | — |
| Exp. | Change | What we learned |
|---|---|---|
| Stage 1 with WavLM features (prototype) | ||
| 01 | Conformer + powerset + global window linking, energy labels | Activity F1 0.83–0.88; only 54–64% of conversations without swaps. |
| 02 | Same model, reference activity instead of predicted (no training) | No swaps at all: the bottleneck is activity, broken by crosstalk in Open Yap’s energy-based labels. |
| 03 | Word-gated labels + profile heads (TS-VAD style) | Test clean F1 0.943, replies 0.902, 97.5% without swaps; on real dialogues on par with pyannote. |
| 16 | Stage-1 heads fine-tuned on a noisy, reverberant chain | No change (noise 0.829 vs. 0.827): the frozen encoder’s features, not the heads, lose replies in noise. |
| 17 | Activity threshold chosen on val | Threshold 0.4: replies +0.006 clean / +0.019 rec at 0.4 dB more leak. |
Speech Separation
Speech separation isolates individual voices from overlapping speech, supporting clearer communication, transcription, and speaker-aware audio processing.
See all 38 papers