Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
Figures & tables
Figure 1: Training with incomplete references. (a) Paired-stem supervision compares predicted audio with known sources; natural recordings usually lack these targets. (b) Our recipe combines audio targets and speaker timelines before adaptation. RL uses model-judge rewards on real recordings. Training details are given in section 4.3 .
Figure 2: Generative separation architecture of Soloist. Continuous audio features and discrete mixture codes condition the backbone. The text head predicts control tokens, and a local transformer generates the codes within each audio frame.
Figure 3: Data construction and the resulting supervision. Known-source mixtures supply audio targets, songs supply teacher pseudo-vocals, and annotated recordings supply speaker timelines. The timeline response contains timestamps and speaker labels.
Input family
Available target
Supervised response
Constructed speech mixtures
Source audio from the construction
Count and per-voice codec sequences
Songs
Teacher-generated vocal audio
One vocal codec sequence with count and delimiters
Natural recordings
Speaker-attributed time intervals
Text-only speaker timeline
Table 1: Training supervision. Timelines supervise text only; pseudo-vocals inherit teacher limitations. Data construction is detailed in section 4.1 .
Figure 4: Judge feedback for a group of candidate separations. Signal checks screen stream validity, copies, duplicates, and count errors. An audio-language model describes clips individually, and a text ranker compares their reports with an anonymous mixture anchor in shuffled and reversed orders. Groups that fail the order-consistency check are dropped. Accepted groups receive shaped rewards, with dead candidates below surviving ones; subtracting the group mean gives the policy advantage.
System
Clean
Noised
SepFormer
0.058
0.218
MossFormer
0.080
0.289
SAM Audio
0.156
0.351
SR-CorrNet
0.165
0.682
AudioSep
0.435
0.568
ConvTasNet
0.493
0.824
Table 2: LibriMix PI-WER ↓ on overlapping speech with the count supplied. Bold/underline: best/second.
Table 4: ViW WER ↓ and DNSMOS [ 53 ] BAK/OVRL ↑ (mean/median). Bold/underline: best/second separator.
SongYT
MIR-1K
MUSDB18-HQ
System
WER
RPA
SingerSim
WER
RPA
SingerSim
RPA
SingerSim
SepFormer
0.780
0.773
0.955
0.848
0.727
0.890
0.462
0.771
MossFormer
0.389
0.847
0.982
0.638
0.783
0.941
0.563
0.845
SAM Audio
0.460
0.846
0.978
0.885
0.865
0.958
0.643
0.909
SR-CorrNet
1.187
0.683
0.742
1.082
0.680
0.728
0.431
0.659
AudioSep
0.388
0.457
0.921
0.455
0.779
0.936
0.366
0.788
Table 5: Singing results: WER ↓ , mean RPA ↑ , and mean SingerSim ↑ . Bold and underline mark the best and second-best comparable results. MUSDB18-HQ has no lyric references. Unprocessed mixtures are omitted: Whisper large-v3 gives very high WER on the original music, so the comparison is not informative.
LibriMix
Recipe
Metric
Clean
Noised
WildTV
SFT w/o timelines
PI-WER
0.139
0.378
0.330
Acc.
0.890
0.574
0.565
Standard SFT
PI-WER
0.140
0.358
0.287
Acc.
0.926
0.735
0.761
RL (Soloist)
PI-WER
0.060
0.192
0.150
Table 6: Speaker-timeline ablation: PI-WER ↓ and count accuracy (Acc.) ↑ . All rows use blind count inference.
Ranking protocol
Groups passing check
Agreement with PI-WER order
Direct ranking from audio (Qwen3-Omni)
0/40
—
Describe, then rank in text (Qwen3.6-27B)
36/40
88.9%
Table 7: Judge validation on 40 LibriMix groups of eight outputs. Agreement is measured on pairs whose PI-WER differs by more than 0.2, excluding pairs the judge ties.
Figure 5: Full-duplex conversation processing. Speaker activity guides window mining and adaptive separation. Voice embeddings link outputs across pieces, and alignment places them on a shared recording timeline. The dashed loop re-queries failed windows and retains improved outputs. Separate identity, missing-speech, and content checks precede transcript-aligned packaging.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Configuration
Backbone
Qwen3-4B [ 45 ]
Continuous encoder
Qwen3-Omni AuT; trainable
Adaptors
Gated MLPs: 2560→8192→2560 for codec embeddings and 2048→8192→2560 for AuT features; separate output-code adaptor
Local transformer
One layer; hidden size 2560; 32 heads of dimension 80; twelve codebooks with 1024 entries plus padding
Codec
MOSS-Audio-Tokenizer-v2; first 12 of 32 residual quantizers; frozen
k1 KL estimator; coefficient 0 in the first phase and 0.001 in the second; no entropy bonus
Optimizer
Adam; learning rate 10−6 in the first phase and 2×10−6 in the second; 10 warm-up updates then constant; (β1,β2)=(0.9,0.98) ; ϵ=10−8 ; no weight decay
Appendix
Table 9: RL configuration for both phases. Only the learning rate and KL coefficient change between phase 1 and phase 2.
Token
Role
<|im_start|> , <|im_end|>
Turn start and turn end / stop
<|vision_start|> , <|vision_end|>
Audio boundaries for input and output
<|vision_pad|>
Input frame, replaced by the mixture embedding
<|video_pad|>
Output frame request, replaced by the generated-code embedding
<|endoftext|>
Sequence padding
Appendix
Table 10: Vocabulary tokens reused for the audio interface.
Component
Setting
Perception
Qwen3-Omni-30B-A3B-Instruct; greedy; maximum 400 tokens; repetition penalty 1.05
Text ranker
Qwen3.6-27B; thinking disabled; temperature 0.3; top- p=0.9 ; maximum 1024 tokens; repetition penalty 1.05
Candidate presentation
M=2 calls; randomly shuffled labels, then reversed order; anonymous mixture anchor included
Tier values (qA,qB,qC,qD)
A/B/C/D ↦ 1.0/0.7/0.4/0.1
Consistency threshold
τmin=0.3 ; accept Kendall’s τb≥τmin
Group rejection
At most one candidate surviving the signal checks; failed mixture or anchor description; fewer than two fully described surviving candidates; failed ranking call; low rank agreement; both calls indistinguishable; conflicting indistinguishable flags
Appendix
Table 11: Judge inference settings; tier scores are averaged over two rankings.
Check
Setting
Stream duration
dk>0.2 s; duration ratio lower bound 0.35, upper bound 1.05
Signal level and activity
RMS >−45 dBFS; active fraction ≥0.05
Activity frames
50 ms; active above −40 dBFS
Log-mel representation
FFT 512; hop 256; 40 mel bands; frame-match tolerance 1.5
Input copy
Frame-match threshold 0.8; per-stream cap 0.1
Duplicate stream
Frame-match threshold 0.8; per-stream cap 0.3
Appendix
Table 12: Signal-gate parameters for reward shaping.
Contemporary a cappella singing often has a lead-and-accompaniment texture, where the lead vocal (Vo) part carries the main melody and the remaining vocal parts provide accompaniment. Owing to their distinct roles, separating the Vo part from the remaining vocal parts, referred to as Vo separation, enables downstream applications such as lyric recognition and minus-one accompaniment generation for vocal ensemble music. Despite these potential applications, acoustic cues for this task are limited because the target and interfering sources are all singing voices with similar acoustic characteristics and often overlap in time, making Vo separation challenging. In this paper, we propose a Vo separation model that uses phoneme alignment of the Vo part as auxiliary information. The proposed model is based on band-split RoPE Transformer (BS-RoFormer), a state-of-the-art music source separation model, and introduces frame-level phoneme labels into its intermediate representations using feature-wise linear modulation (FiLM). Experimental results show that phoneme-alignment conditioning improves Vo separation performance over an audio-only baseline and yields larger average gains than conditioning only on Vo singing/silence activity. Further analysis suggests that the advantage of phoneme-label information is larger when fewer remaining vocal parts share the same phoneme as Vo.
Yuma Narahata, Tomohiko Nakamura, Yuki Saito +1
The University of Tokyo, Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
Yutong Wen, Ke Chen, Prem Seetharaman +5
Adobe Research, USA · Massachusetts Institute of Technology, USA
Music source separation systems typically extract a single vocal track and do not distinguish between multiple singers. We study singer-informed vocal source separation for multi-singer mixtures. Our framework introduces a short enrollment recording of a target singer to guide separation through a learned embedding. The singer embedding is incorporated using feature concatenation or feature-wise linear modulation (FiLM), enabling the model to focus on the target singer while suppressing interference. We construct a duet dataset based on DAMP-VSEP with quality filtering and non-overlapping enrollment segments. Experiments on solo and duet settings show that while baseline models perform well for single-singer mixtures, the proposed method improves target-singer extraction in multi-singer cases, increasing target-singer SI-SDR from 0.33 dB to 5.58 dB. Fréchet Audio Distance (FAD) further shows improved perceptual quality and better alignment with target audio distributions. Code and checkpoints are available at https://github.com/jocelynxu01/singer-separation-paper.
Jocelyn Xu, Minje Kim
University of Illinois Urbana-Champaign, Siebel School of Computing and Data Science