cs.SDOct 26, 2026

The Voice on the Line

Authors: Husein Zolkepli

Organizations: Scicom (MSC) Berhad, Malaysia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A transcriber listening through one microphone also hears everyone near it. We train causal voice isolators: masking transformers of 140M, 243M and 374M parameters, 20 ms of latency, enrolling the caller from the call itself. Short evaluation windows hide two failures that real calls expose: forgetting the caller when another voice talks alone for a while, and damaging a caller who is alone. Long calls with turns, memory beyond the attention window, real headset crosstalk and drifting caller levels address both, but each trades one against the other. WavLM-Large's own weights, made causal and trained with the model, move the trade: M with them (555M parameters) lowers the error on our development sets from 11.44% to 10.55%; a student trained from scratch to imitate WavLM does not, and w2v-BERT 2.0 and Whisper's encoder, pre-trained on far more audio, help less or not at all. A lone caller loses most at the start of a recording: no training call opened with someone already talking. Training on windows cut from longer calls lowers the error on clean development calls from 3.92% to 2.96%; this is our release. Live, the release makes 16.13% word errors on 60 s AMI windows, against 23.66% for VoxLock, the strongest open isolator, and 10.19% on its own AMI windows, against 11.71%; it is level on Krisp's benchmark (5.74% against 5.63%), and a caller who is alone still loses more words than with VoxLock (5.58% against 2.92% on clean test calls). Distilled into 22.1M parameters without WavLM, it runs live on one CPU thread inside a LiveKit agent, at 33% of a core per call in 20 ms frames, with 10.66% error on our development sets (the release 10.49%). We open-source the release and this CPU model, with a LiveKit plugin, at https://huggingface.co/collections/Scicom-intl/voice-isolator.

Figures & tables

Figure 1: Krisp: word error rate against leaked words per minute of background-only speech, one point per system, our models live. A noise suppressor (DeepFilterNet3) keeps the other voices; isolators remove them.
Figure 2: How a training example is made. Blue: the primary talker; amber: synthetic crosstalk; grey: noise; purple: the enrollment. The device filters and band limit act on mixture and target alike; μ\mu -law, codecs and clipping on the mixture only. Every probability is in Appendix C .
Figure 3: Training speech by source. Common Voice ( Ardila et al., 2020 ) and Multilingual LibriSpeech ( Pratap et al., 2020 ) are restored to studio quality with Sidon ( Nakata et al., 2025 ) ; the rest is recorded clean.
Figure 4: Noise by kind. One clip in 20 is held out for development and tests.
Figure 5: Reverberation time of the simulated rooms, by type. Car cabins are 3.3% of rooms: talker placement fails more often in a cabin and the room is redrawn.
Figure 6: Distance from each talker to the microphone. The primary sits at a headset, a handset or a speakerphone; the background talkers are further away, but not always (a tenth of them are rendered dry, as close as the primary).
Figure 7: One simulated call, 8 s, built step by step; panels 2 to 5 add one stage each to the one before, panel 6 is what the model is asked to output. Bars under a spectrogram: blue where the primary talks, amber where only a background talker does (where a leaked word would count). Held-out speakers, noise and room; same colour scale throughout.
Figure 8: How hard the simulator makes it: the lowest signal-to-interference ratio among an example’s background talkers, and the lowest signal-to-noise ratio among its noises, both relative to the primary talker.
Figure 9: The line: band limits (green) and real codecs run through ffmpeg on the mixture (olive). G.723.1 and Codec2 are kept out of training for testing.
Figure 10: Five simulated calls of the long-call stage (development speakers). Blue: the caller talks; amber: only the background voice does. The caller often comes back after the other voice has talked alone for several seconds.
Figure 11: Share of each set’s background-only time in stretches longer than a given length (gaps under 1 s closed). Every VoxLock-style window set stops at 4 s.
Figure 12: The realtime model. One token per 10 ms frame; the speaker vector enters every layer through FiLM. The release adds a causal WavLM-Large whose layers are mixed into the tokens.
Figure 13: The Sidon-architecture models: inference in 4 s windows (top) and the two training signals (bottom).
Figure 14: The release recipe. AdamW, cosine schedules, bfloat16, weights averaged (0.999). The long-call stage widens the attention window and adds the frame LSTM with its output at zero, so it starts as the model before it.
Figure 15: Every test set: word error rate of the release (M + WavLM), live, minus VoxLock live, with a 95% interval over callers (an arrow: the interval runs past the axis). Left of zero, we make fewer errors. AliMeeting: character error rate. Calls, caller alone: change against plain recognition.
Figure 16: Krisp: background let through, by how long the background-only stretch has lasted, relative to each system’s gain while the caller talks. Our S before and after the long-call stage; Krisp is never used to choose.
Figure 17: The long-call stage, S, 4,000 steps from one checkpoint, one change per bar. Short examples that end with the other voice alone (second bar) also cut the caller when they come back.
Figure 18: S through the training stages, live enrollment: the three meeting development sets (mean) and clean real calls, where any change is damage to a lone caller. Dashed: VoxLock live.
Figure 19: Development sets, every system live. Down: less of the other voices in meetings. Left: less damage to a lone real caller. Stars: M with WavLM inside.
Figure 20: Development calls with the caller alone: share of the caller’s speech removed (output 20 dB or more under the input), in the first 3 s of speech, in the 5 s after a silence of 3 s or more, and elsewhere. Ours before the window cut, and the release.
Figure 21: One development call, the caller alone: how much of the caller each system keeps, per 100 ms of speech. After a stretch of digital silence (grey) the release removes the returning caller for several seconds, as the model before the window cut does.
Figure 22: Development calls with the caller alone: share of the caller’s speech removed against the time since the recording’s first speech (bins, plotted at their middle). M + WavLM from a cold start, with the caller’s enrollment from the first frame, and after a second of digital silence; the release, trained on windows cut from longer calls.
Figure 23: Every fix tried for the lone caller, from M + WavLM before the window cut: rules at inference and changes to training, word error rate on the clean development calls against dev-4. Fixes whose dev-4 lies above the axis sit on its top edge with their dev-4.
Figure 24: Where each fix keeps the lone caller: share of the caller’s speech removed in the first 3 s of speech and in the 5 s after a silence of 3 s, development calls.
Figure 25: The window cut in the long-call stage, two seeds at S and at M: word error rate on the clean development calls against the 60 s AMI development windows, an arrow from each run without the cut to the same seed with it, and two more changes on top of the cut at M. Labels: dev-4 without and with the cut.
Figure 26: Stage 1 of S, 30,000 steps, one change per bar: mean word error rate of AMI dev and NOTSOFAR dev. These short windows cannot see long calls or a lone caller.
Figure 27: Enrollment modes of the S release candidate: meetings (mean of three development sets) and a lone caller (clean calls). Only the live modes can run in a call.
Figure 28: Dev-4 (live enrollment) against parameters: S, M and L with the release recipe, alone and with WavLM inside. Dashed: VoxLock.
Figure 29: How close a causal encoder gets to WavLM-Large (cosine to its layer 16, held-out simulated calls) during distillation: a student from scratch, and WavLM’s own weights made causal (the dot: before any training).
Figure 30: Dev-4 of M with each use of WavLM, live enrollment (the selection criterion: three meeting sets and clean calls). Dashed: VoxLock.
Figure 31: Dev-4 of M with each pre-trained encoder made causal and trained with the model, against the encoder’s size. Filled: the long-call stage only (4k steps); open: from the real-crosstalk fine-tune on (12k steps).
Figure 32: Krisp: leaked words per minute of background-only speech. The Sidon-architecture models are trained on the same calls as ours.
Figure 33: Simulated stress sets: SI-SDR improvement of our M and VoxLock’s PyTorch script, both given the set’s enrollment clip. Scale-invariant, so output gain does not matter.
Figure 34: The server. Open calls stay in the first rows of the state bank, so the step reads and writes it in place.
Figure 35: Step time (99th percentile) against concurrent calls, fp16 with CUDA graphs, 20 ms chunks, one GPU, release architectures. Below the dashed line a step keeps up with real time.
Figure 36: End to end over WebSockets on one GPU, live enrollment, 20 ms chunks: the delay from a chunk leaving the client to its isolated audio coming back (99th percentile solid, median dashed), over the chunks sent after each call’s first 15 s, against live calls; M and the release, with every connection in one front-end process or spread over eight. Runs where calls timed out are left out. One process is erratic once it falls behind: a later, larger run can land on time again.
Figure 37: The serving kernels at 64 calls of the release: each part’s time as a share of what it took before them, the milliseconds on the bars.
Figure 38: A fresh server given 64 calls of the release at once on 20 ms chunks: the worst 99th percentile of a chunk’s round trip in its first 30 s (*: in its first minute), as the live enrollment’s cost was taken out of the step.
Figure 39: Thirty minutes at 64 calls of the release on 20 ms chunks: a chunk’s round trip in each minute.
Figure 40: The isolator in a LiveKit agent: a frame processor on the caller’s audio stream, ahead of the VAD, the turn detector and the recogniser. Each call keeps its own state; the onnxruntime session is shared by every call in the process.
Figure 41: What one 10 ms hop sees, to scale over a 20 s call, while LiveKit hands the call over in 50 ms frames: the attention window, the LSTMs and the speaker vector belong to the call, not to the frame.
Figure 42: One call with another voice: LiveKit’s smoothed Silero probability on the raw call, behind GTCRN and behind the release. Green: the caller talks (the same VAD on the caller’s own channel); orange: only the other voice does. Above the dashed line a window counts as speech.
Figure 43: LiveKit’s VAD behind each system: speech starts inside the labelled stretches where only another voice talks, per minute of them, on three sets of real recordings.
Figure 44: The CPU student beside the release it learns from. The pipeline is the same. The band encoder and decoder, with their band LSTMs, the mask head and the enrollment encoder keep their shapes and start from the release’s weights (92 tensors); the transformer shrinks (C22: 8 layers of width 384; C10: 6 of 256; C51: 12 of 512) but keeps the 10 s window and the frame LSTM; WavLM is dropped. On every training call the release’s output is a second target, beside the clean caller.
Figure 45: Distilling the release into C22, the release recipe every time: dev-4 from scratch, started from the release’s weights, with the release’s output as a second target, and both. Dashed: VoxLock live and the release.
Figure 46: Dev-4 against the share of one CPU thread a live call takes through the plugin in 20 ms frames, for each distilled size in float and in 8 bits, and for VoxLock’s own 8-bit package. Dashed: the release, on a GPU.
Figure 47: The share of one CPU thread a live call takes, by the frame size LiveKit hands the agent: GTCRN (noise only), the CPU student in float and in 8 bits, and VoxLock’s default 8-bit model, each through its own streaming entry point on the same calls, timed in one run. Above the dashed line a call needs more than one thread.
Figure 48: Every test set: word error rate of the CPU student, live, on one CPU thread, minus VoxLock live, with a 95% interval over callers. Left of zero: the student makes fewer errors.
Figure 49: One call through a running agent, in time. The caller stops at 0.8 s while another voice goes on. LiveKit’s VAD ends a turn after 0.55 s of silence: on the raw frames it hears the other voice and waits for it too; behind the isolator it hears only the caller, late by the isolator’s delay (10 ms on the CPU, 40 ms through the GPU server). The end of the turn is where the agent’s turn detector lets it answer. A schematic, with LiveKit’s rule and the measured delays.
More in this topic

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

Explore similar work

CardsList
Oct 14, 2026

Soloist: A Generative Model for Blind Voice Separation in the Wild

Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.

Chen Yang, ChuFan Yu, Botian Jiang +5

Fudan University · Shanghai Innovation Institute

Audio Source SeparationSpeech Separation
AskPDF
Aug 28, 2026

Auditing generative audio calls for known-task audio-llm evaluation

Speech and audio LLMs are evaluated by comparing waveform predictions with predictions from an automatic speech recognition (ASR) transcript. For fixed closed-set tasks, this conflates acoustic evidence with the need to invoke a generative audio model. We estimate incremental call value with matched selectors sharing pre-call evidence. Each policy may retain the transcript label, use a local encoder, or invoke a generative model; matched control removes generative actions but preserves pre-call evidence and development selection. On VocalSound, transcript-only accuracy is 0.296, while supervised CLAP and WavLM controls reach 0.850 and 0.854 without calls. Full selector reaches 0.925 at 12.5% calls versus 0.921 for matched No-call selector (difference 0.004; 95% CI [-0.025, 0.033]). Thus, results do not show a call gain after transcript and encoder evidence are available. Relevant quantity is incremental accuracy from allowing calls, not the waveform-transcript gap.

Mengzhe Geng

National Research Council Canada

Audio-Language Model EvaluationAudio Classification
AskPDF
Jun 7, 2026

From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data

We present a voice conversion (VC) framework that utilizes K-Nearest Neighbors (KNN) retrieval over WavLM representations to align non-parallel source and target speech, constructing synthetic training pairs for supervised learning. The retrieved segments serve as synthetic inputs, while real target audio provides ground-truth outputs, forming a synthetic-to-real training paradigm that naturally supports multilingual data without requiring parallel corpora or explicit alignment. To ensure consistent target-speaker identity, we incorporate a speaker loss derived from a pretrained speaker verification model. Experiments across multiple languages demonstrate that the proposed approach achieves high naturalness and strong speaker similarity, outperforming competitive VC baselines, despite being trained exclusively on English data. Samples can be accessed at: https://palindromic-vc.github.io.

Moshe Mandel, Shlomo E. Chazan

Independent, Israel · OriginAI, Israel

Zero-Shot Voice ConversionSpeech ProcessingVoice ConversionCross-Lingual Transfer
AskPDF