cs.SDSep 17, 2026

Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation

Authors: Bertil Braun

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Natural spoken interaction requires more than streaming ASR, language generation, and speech synthesis: a system must react to overlap without canceling on every acknowledgment, prepare a response before a turn is certain, and ensure canceled audio cannot enter conversation history. We present Voice-Light, a full-duplex cascaded voice agent that combines immediate acoustic onset, a causal adapter sharing a streaming ASR encoder, reversible playback control, and private speculative response generation. Structured tool calls execute concurrently with audible bridge speech, while browser acknowledgments make rendered audio authoritative for durable history. Locked evaluation on 1,673 real-conversation silence candidates found that an earlier learned completion checkpoint preserved a 2.70% false-cutoff rate but reached only 12.53% end-of-turn recall, compared with 95.60% for a Silero timing policy. The deployed system therefore retains a hybrid controller rather than claiming a learned-policy replacement. Across three unscripted operator-run microphone sessions, 36 measured response turns had a 758 ms median from final VAD endpoint to first server audio; 21 turns were below 800 ms. These sessions are an instrumented case study, not a controlled user evaluation. We release the synthetic data, model artifacts, evaluation code and summaries, source code, and deployment configuration supporting the result.

Figures & tables

Figure 1: Production runtime. Immediate acoustic reaction and learned interaction evidence meet at a typed controller. Tool calls and results use a structured loop; acknowledged browser playback is authoritative for durable assistant history.
Figure 2: Two model-training branches feed one evaluated deployment. The turn-taking branch combines synthetic interaction timelines with separately prepared human evidence; the language-model branch uses synthetic typed conversations.
MaterialConversationsHours
MagicHub82.77
TurnBench377.31
Additional authorized material6226.22
Total10736.30
Table 1: Human conversational material in the locked corpus manifest. Source audio remains subject to its original access terms; only derived manifests and code are distributed by the project.
Checkpoint (step)Role and evidence
Synthetic (2,250)Pretraining seed; synthetic and human validation
Historical (3,500)V1 locked test and V2 validation
Human fine-tune (750)Deployed adapter; human validation and runtime only
Completion challenger (625)Later experiment; V2 validation only
Table 2: Turn-adapter checkpoint lineage. Parenthesized values are training steps. “Validation” and “test” name distinct locked inventories; evidence is not transferred between artifacts.
BehaviorBaseFine-tuned
Correct tool decision80.0%95.2%
Exact tool name78.6%94.3%
Valid argument schema80.0%95.2%
Concise 1–12-word bridge7.6%94.3%
Correctly avoided a tool70.0%100.0%
Valid post-tool continuation98.3%100.0%
Table 3: Balanced synthetic holdout for Qwen3-1.7B. The same 160 states were decoded with three seeds. Expected calls, no-tool states, and continuations contribute 210, 90, and 180 repeated predictions respectively; percentages therefore do not represent independent tasks.
Figure 3: Recorded assistant-token validation loss after each logical pass. The directly comparable zero-update loss was 4.5398 and is omitted to preserve resolution. Loss bottomed after pass 4, then rose even while free-generation protocol metrics improved.
MetricVLSileroSmartLiveKit
False cutoff2.70%2.70%13.51%2.70%
HOLD errors ( n=37n=37 )1151
EOT recall12.53%95.60%20.72%91.50%
Mean latency770 ms656 ms684 ms654 ms
Threshold0.900.050.950.15
Minimum delay560 ms640 ms80 ms640 ms
Table 4: Locked real-conversation test under the original V1 yield-target protocol. Mean latency includes timeout actions. VL is historical Voice-Light step 3,500; policy values are threshold, minimum delay, and timeout. Possible training-source overlap makes Smart Turn contextual only.
Figure 4: Observable activity during a 20-second excerpt from an interactive microphone session. Green spans show detected user speech and gold spans show acknowledged assistant playback. The session exercised short backchannels and a floor-taking interruption, but the lanes intentionally do not assign semantic labels to individual overlaps.
MeasurementObserved value
Sessions / measured turns3 / 36
Final VAD to first server audio, median758 ms
Final VAD to first server audio, mean932 ms
Final VAD to first server audio, range528–1,652 ms
Turns below 800 ms21 / 36
Fully traced turns13
Table 5: Deployment case-study measurements. Aggregate response latency covers three sessions; component medians come from the 13-turn fully exported trace and overlap in time.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ArtifactRevision, hash, or recorded result
Report sourceCommitted TeX, figure assets, and build instructions in the source repository
Runtime source snapshote2f79adc on master ; evaluated two-GPU deployment d78115d2
Qwen3-4B-Instruct-2507cdbee75f17c01a7cc42f958dc650907174af0554
Qwen3-0.6B summarizerc1899de289a04d12100db370d81485cdf75e47ca
Nemotron Streaming 0.6Bebe59e5a817142986528bbbee5dba8db7b38ed50
Kyutai TTS 1.6Bf65439609986c392cb12df63938abcc550c3fb15
Appendix
Table 6: Principal revisions, hashes, and recorded validation for the final runtime. Full commands and additional hashes are recorded in the repository runbooks [ 26 ] .
More in this topic

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 136 papers

Explore similar work

CardsList
Oct 15, 2026

VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model

A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.

Yang Xiao

The University of Melbourne

Turn-Taking PredictionSpeech Language ModelsVoice Agent Evaluation
AskPDF
Mar 9, 2026

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.

Shangeth Rajaa

Anyreach AI

Speech Language ModelsTurn-Taking PredictionSpeech Foundation ModelsFull-Duplex Spoken Dialogue Systems
AskPDF
Sep 25, 2026

RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue

Many voice interaction applications require exact control over both the content and delivery of responses, typically using pre-recorded lines. Recent full-duplex models respond with low latency but cannot guarantee exact content or reproduce a specific recorded performance, while cascaded systems can be constrained to predefined responses at the cost of additional latency. We propose RePlay, a spoken dialogue system adapted from PersonaPlex that handles multi-turn conversations by retrieving and playing pre-recorded lines. Using probing, we identify the layer and frame at which the upcoming response becomes recoverable, and use this hidden state as the retrieval query. RePlay retains only the layers up to that point and replaces text and speech generation with lightweight turn-taking and retrieval heads. In simulated multi-turn interviews, RePlay reaches a median latency of 383 ms, 3 to 7 times lower than ASR-LLM cascades of comparable dialogue quality, at the cost of lower exact-line accuracy. In a user study, participants preferred RePlay in 63% of ratings versus 12% for a fast cascade with a small LLM (p = 0.008), and showed a non-significant preference (46% vs. 21%) over a slower cascade with a stronger LLM.

Sathvik Udupa, Naveen Kumar, Ryan Folmsbee

Disney Research Imagineering, Los Angeles, USA

Spoken Dialogue SystemsMulti-Turn Dialogue ManagementSpeech Language ModelsFull-Duplex Spoken Dialogue Systems
AskPDF