We present a novel approach to the problem of combining two voice-line channels of dyadic conversational audio into a unified conversation with controllable back-channeling and turn-taking dynamics. In voice performances for the game industry, the two tracks of voice-lines are typically recorded or synthesized in isolation. Our method takes two such unsynchronized tracks and uses a neural scheduler to sequence them into a single dynamic conversation, modifying both tracks by inserting or removing pauses and introducing voice overlap as appropriate. We further propose a back-channel predictor that is able to predict opportunity for back-channels for a given conversational turn's audio and content, which the scheduler can then incorporate. We evaluate our scheduler and predictor independently and in combination by: various quantitative comparison; a qualitative and perceptual comparison to prior art and naive baselines; a compelling set of results showing animator control over the diversity, extent and nature of back-channeling and turn dynamics.
Figures & tables
Figure 1. Input turn-clips are processed by the Back-channel Predictor, which determines and synthesizes a candidate verbal back-channel. The Neural Scheduler then sequences the turn-clips and generated back-channels with appropriate temporal gaps or overlap into a conversational dialogue. Circled numbers indicate animator-centric parameters: (1) turn-clip splitting, and (2) a back-channel probability threshold to control back-channel frequency; (3) a back-channel vocabulary with varying valence, duration, and intensity; and (4) timing constraints to control pause and overlap in conversational turn-taking. Pipeline diagram. Input voice clips from speakers A and B pass through the Back-channel Predictor, which adds a generated back-channel clip, and then the Neural Scheduler, which arranges all clips on a two-channel timeline. Four insets show the animator controls: clip splitting, the back-channel probability threshold, the back-channel vocabulary, and timing constraints.
Model
F1 ↑
P ↑
R ↑
AUC ↑
EER ↓
Baselines
Wang et al.
0.43
0.46
0.40
0.73
0.33
Lin et al.
0.42
0.34
0.55
0.69
0.36
Ours
Full model
0.51
0.53
0.49
0.80
0.28
w/o audio
0.34
0.55
0.25
0.75
0.32
Table 1. Binary back-channel classification results on held-out data. Higher F1, P, R, AUC are better; lower EER is better. Table of F1, precision, recall, AUC and EER for two baseline classifiers and for our full model with four ablations. Our full model has the best F1, recall, AUC and EER.
Class
F1 ↑
P ↑
R ↑
AUC ↑
EER ↓
Support
No-back-channel
0.89
0.88
0.90
0.80
0.28
8,891
Acknowledgment
0.44
0.46
0.43
0.79
0.28
1,709
Reactive
0.23
0.24
0.22
0.76
0.31
400
Table 2. Per-class performance of our full model on the three-way merged task (No-back-channel / Acknowledgment / Reactive). Table of F1, precision, recall, AUC, EER and support for each of the three classes. No-back-channel scores highest and Reactive lowest.
Figure 2. Illustration of how voice-line scheduling can be formulated as a next-token prediction problem. Diagram of the token layout. The input region holds each speaker's voice lines as separate token sequences with start and end turn markers and silence separators; the output region holds two frame-aligned channels, A and B, in which speech and silence tokens encode the timing.
Figure 3. Illustration of positional id and channel id assignment for the corresponding embeddings. A token sequence with two rows of numbers beneath it. Input tokens have positional ids 1 to 10 and channel id 0; each pair of output tokens shares one positional id (11, 12, 13) and has channel ids 1 and 2.
Figure 4. Between-speaker intervals (negative: overlap; positive: gap) in generated vs. ground-truth dialogue. Without duration prediction (left), generated intervals collapse into a spike near zero; with it (right), they track ground truth. Two histograms of between-speaker intervals from -2000 to 2000 ms, comparing generated and ground-truth dialogue. With the next-token objective only, generated intervals collapse into a single spike just above zero; with the added silence prediction objective, the generated distribution closely follows the ground truth.
Cumulated duration /min
Model
∣ΔIPU∣
∣ΔPause∣
∣ΔGap∣
∣ΔOverlap∣
Naive Cascade
4.3
5.5
0.9
3.6
dGSLM
4.6
3.6
1.8
1.9
LSLM
4.1
3.4
1.5
2.3
NTPP
3.3
2.8
1.4
1.9
Moshi 0.8
16.0
6.8
8.3
2.1
Table 3. Turn-taking statistics (cumulated duration/min) for generated dialogues, compared to ground truth using mean absolute differences ( ∣Δ∣ ) between predicted and ground truth values. Lower ∣Δ∣ values indicate better performance, with the best results highlighted in bold. Table of absolute differences from ground truth in IPU, pause, gap and overlap duration per minute for eight models. The Neural Scheduler has the smallest difference on three of the four metrics.
Figure 5. An illustrative example of the three passes. Pass 1 places the turns, Pass 2 inserts the back-channel b with a brief pause in A’s turn (dotted), and Pass 3 re-times when A resumes. Dashed boxes mark each clip’s search range. Three stacked two-channel timelines over 80 frames. After Pass 1, turn A1 is followed by the reply B1 with a slight overlap; after Pass 2, a back-channel b is placed under A1, which is split by a short pause; after Pass 3, the second part of A1 and the reply B1 are shifted later.
Back-channel
Model
TOR ( ↓ )
Freq ( ↑ )
JSD ( ↓ )
Freeze-Omni
0.636
0.001
0.997
Gemini
0.091
0.012
0.896
Moshi
1.000
0.001
0.957
dGSLM
0.691
0.015
0.934
PersonaPlex
0.327
0.025
0.649
Table 4. Full-Duplex Bench back-channel evaluation results. Table of turn-override rate, frequency and Jensen-Shannon divergence for back-channels from six systems. Our system has the best value on all three.
Figure 6. Voice activity timelines for two dialogue examples comparing back-channels generated by PersonaPlex and our proposed model. Two voice activity timelines, Examples A and B, each showing a speaker's turns together with the back-channels produced by our model and by PersonaPlex. Our back-channels are short and fall within the speaker's turns, while PersonaPlex's are long or come after the speaker has finished.
Figure 7. Voice activity timelines for one dialogue, comparing the turn scheduling of the naive pause-based baseline (top) with that of our proposed model (bottom). Voice activity timeline over 20 seconds comparing a naive fixed-pause baseline (top) with our scheduler (bottom) on the same dialogue. Highlighted regions show that our schedule contains overlapping speech and gaps of varied length, whereas the baseline's gaps are uniform.
Figure 8. Four axes of animator control: (1) back-channel content, by swapping candidates; (2) back-channel frequency, by lowering the prediction threshold; (3) speech overlap, by tightening onset constraints; and (4) thinking pauses, by widening one speaker’s search window. Four panels of two-channel voice activity timelines, one per control. Panel 1 swaps back-channel candidates from none to brief to long; panel 2 lowers the threshold to go from zero to four back-channels; panel 3 tightens constraints to increase overlap; panel 4 widens gaps to add thinking pauses for speaker A or B.
Backchannels, brief acknowledgements like "uh-huh" produced while the other party may still be talking, are central to natural conversation, but full-duplex spoken dialogue models rarely model them explicitly. We introduce a lightweight backchannel head that predicts, from a full-duplex model's own hidden states, when a backchannel should begin. Once this probability crosses a tunable threshold, a backchannel is force-decoded. Attached to both a 7B (PersonaPlex) and a 1B (F-Actor) model, it generalizes across scale. Probing confirms the hidden states anticipate real human timing, and generation evaluation shows more frequent, better-timed backchannels. Human raters judge the resulting backchannels on par with real ones.
Maike Züfle, Peter Polák, Sefik Emre Eskimez +3
Karlsruhe Institute of Technology, Germany · AppTek, Germany · Charles University, Czech Republic +2
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
Dyadic conversational motion generation is essential for realistic interactive digital humans. Existing approaches typically model conversational behaviors within unified dyadic generators. However, such holistic formulations tend to couple self-speech-driven motion with partner-responsive social feedback, leaving the interaction-specific component implicit and underutilizing the speech-motion correspondence already learned by pretrained monologic motion models. We propose Learn2Chat, a unified framework that models dyadic motion as interaction modulation over pretrained monologic motion priors. This design separates intrinsic speech-driven motion from social interaction effects and enables more structured interaction modeling. Specifically, we introduce a Monologic-Anchored Motion Factorization scheme that leverages the semantic motion manifold learned from monologic data to disentangle audio-driven motion dynamics from interaction-induced modulation, yielding clean interaction representations from dyadic sequences. On top of this representation space, a Cross-Attentive Interaction Latent Prediction module maps paired speech signals to interaction latents through cross-branch attention and interaction alignment. During inference, the predicted interaction latents modulate canonical monologic motion to generate coherent and synchronized dyadic behaviors in a data-efficient manner. Extensive experiments on the DualTalk benchmark demonstrate that Learn2Chat achieves state-of-the-art performance across both quantitative metrics and perceptual evaluations. Moreover, the framework is model-agnostic and seamlessly integrates with diverse pretrained monologic motion backbones, highlighting the effectiveness of prior reuse and interaction adaptation for scalable conversational motion generation. More visual results are available on the project page.
Zikai Huang, Siyue Chen, Xuemiao Xu +4
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2