Many voice interaction applications require exact control over both the content and delivery of responses, typically using pre-recorded lines. Recent full-duplex models respond with low latency but cannot guarantee exact content or reproduce a specific recorded performance, while cascaded systems can be constrained to predefined responses at the cost of additional latency. We propose RePlay, a spoken dialogue system adapted from PersonaPlex that handles multi-turn conversations by retrieving and playing pre-recorded lines. Using probing, we identify the layer and frame at which the upcoming response becomes recoverable, and use this hidden state as the retrieval query. RePlay retains only the layers up to that point and replaces text and speech generation with lightweight turn-taking and retrieval heads. In simulated multi-turn interviews, RePlay reaches a median latency of 383 ms, 3 to 7 times lower than ASR-LLM cascades of comparable dialogue quality, at the cost of lower exact-line accuracy. In a user study, participants preferred RePlay in 63% of ratings versus 12% for a fast cascade with a small LLM (p = 0.008), and showed a non-significant preference (46% vs. 21%) over a slower cascade with a stronger LLM.
Figures & tables
Figure 1: Per-layer probing for the upcoming response: (a) QA accuracy, (b) clustering accuracy. Dotted line: layer 15.
Figure 2: RePlay model architecture
Figure 3: Nobu, our agent: human-like but distinctly non-human, to keep user expectations flexible.
#
ASR
Turn end
Response selection
A
Azure
ASR final
GPT-4.1 picks from the full line set
B
Kyutai [ 30 ]
VAD ≥0.6
Gemma-4-E4B picks from the full line set
C
Kyutai [ 30 ]
VAD ≥0.6
Gemma-3-1B free reply, mapped to the nearest line (nomic-embed [ 16 ] )
Table 1: Baseline systems.
Retrieval
Dialogue quality
Turn-taking
Latency (ms)
System
R@1 ↑
R@3 ↑
R@10 ↑
Q ↑
IntentAgr ↑
CR (%) ↓
p10 ↓
p50 ↓
p90 ↓
mean ↓
Cascade A
0.52
0.62
0.71
3.90 ± 0.33
0.85 ± 0.16
0.2
2303
2612
3012
2652
Cascade B
0.36
0.55
0.62
3.77 ± 0.45
0.77 ± 0.19
8.7
780
1226
1414
1145
Cascade C
0.17
0.27
0.34
2.99 ± 0.63
0.35 ± 0.19
17.0
296
591
784
569
RePlay (ours)
0.35
0.50
0.60
3.76 ± 0.43
0.83 ± 0.16
3.6
333
383
439
387
Table 2: Online evaluation with a simulated interviewer (100 dialogues per system). Metrics as in Sec. 4.4 ; Q: judge rating (1–5) for engagement and responsiveness; ± : std. over dialogues. Latency excludes cutoff turns. Best per column highlighted.
D1
D2
R@1
R@3
R@10
Q
IntentAgr
Training stages
✓
–
0.18
0.28
0.37
3.16 ± 0.53
0.45 ± 0.18
–
AS
0.23
0.43
0.54
3.72 ± 0.45
0.77 ± 0.17
✓
AS
0.35
0.50
0.60
3.76 ± 0.43
0.83 ± 0.16
Generalisation (with D1)
✓
UL
0.28
0.46
0.57
3.74 ± 0.44
0.76 ± 0.17
Table 3: Ablation on training data. D1: synthetic pretraining. D2: 2 h fine-tuning on Nobu with all lines seen (AS), held-out lines (UL), or held-out intents (UI). Metrics as in Table 2 .
Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path - a duplex S2S model - speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency response onset, while continuing to monitor live audio events. The slow path - a cascaded ASR -> LLM pipeline - generates a higher-quality continuation conditioned on the committed prefix, producing an uninterrupted utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the cascaded pipeline. With GPT-4.1 as the back-end, RelayS2S substantially reduces response latency while preserving nearly all of the cascaded pipeline's textual quality. On synthetic voice dialogues, it achieves a P90 first-chunk latency of 81 ms, excluding TTS and network latency, compared with 1,006 ms for the cascaded baseline. On real voice dialogues, RelayS2S reduces average first-chunk latency by 479 ms while retaining 99% of the cascaded pipeline's textual quality. These benefits become larger as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code is publicly available at: https://github.com/mailong25/relays2s
Long Mai, Junli Liang
Trinity College Dublin, Ireland · University College Dublin, Ireland
Natural spoken interaction requires more than streaming ASR, language generation, and speech synthesis: a system must react to overlap without canceling on every acknowledgment, prepare a response before a turn is certain, and ensure canceled audio cannot enter conversation history. We present Voice-Light, a full-duplex cascaded voice agent that combines immediate acoustic onset, a causal adapter sharing a streaming ASR encoder, reversible playback control, and private speculative response generation. Structured tool calls execute concurrently with audible bridge speech, while browser acknowledgments make rendered audio authoritative for durable history. Locked evaluation on 1,673 real-conversation silence candidates found that an earlier learned completion checkpoint preserved a 2.70% false-cutoff rate but reached only 12.53% end-of-turn recall, compared with 95.60% for a Silero timing policy. The deployed system therefore retains a hybrid controller rather than claiming a learned-policy replacement. Across three unscripted operator-run microphone sessions, 36 measured response turns had a 758 ms median from final VAD endpoint to first server audio; 21 turns were below 800 ms. These sessions are an instrumented case study, not a controlled user evaluation. We release the synthetic data, model artifacts, evaluation code and summaries, source code, and deployment configuration supporting the result.
LLM-based full-duplex voice services allow users to speak while the assistant is responding. Because servers can generate output and advance dialogue state faster than clients can play it, subsequent user speech may be interpreted based on content the user never heard. We call this failure Generative Context Mis-anchoring (GCM). To address GCM issues, we present PACE, a provider-independent middleware layer that anchors model-facing context to the client playback boundary, a system-observable proxy for what the user could have heard. After an interruption, PACE repairs this context to exclude assistant content that never reached playback, while preserving low-latency generation across heterogeneous voice runtimes. We implement PACE's audio-only projection path end to end in a browser-based realtime voice assistant using a black-box speech model, without modifying the model service. We also construct GCM-Bench, a new controlled benchmark dataset of 108 playback-relative referent-anchoring cases. On GCM-Bench, PACE raises Referent Anchoring Accuracy from 25.0% to 96.3% over a cancellation-only baseline. On 200 Full-Duplex-Bench v1 interruption samples, it preserves interruption response quality. These results show that grounding model-facing context in actual playback is a practical way to maintain consistency in full-duplex voice dialogue.