Real-Time Short-Response Selection for Spoken Dialogue Systems
Authors: Hien Ohnaka, Kazuyo Onishi, Koichiro Yoshino
Organizations: Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama-cho, Toyonaka, Osaka, 560-8531, Japan · Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, Nara, 630-0101, Japan
To realize spoken dialogue systems that can interact naturally with users, it is important for the system to respond to user utterances at appropriate timings. To achieve such natural turn-taking, the system needs not only to predict appropriate response timings but also to mask delays caused by language generation and speech synthesis so that they are not perceived by the user. We propose a real-time short-response selection task that can be integrated with existing turn-taking models to mask generation delay. To address this task, the proposed model learns the mapping between dialogue context and short responses using contrastive learning. When a turn-taking event is detected, the system selects a response from a set of pre-generated candidates. The selected response is played while the main response is being generated. Experimental results show that the proposed method learns to select short responses based on dialogue context, outperforming random and LLM-based baselines. Moreover, integrating response selection maintained turn-taking prediction performance and added minimal processing overhead.
Figures & tables
Figure 1: Overview of the proposed real-time short-response selection.
Figure 2: Original VAP architecture used to initialize the turn-taking backbone. The contrastive predictive coding (CPC) encoder [ 32 ] is used as the feature extractor that converts each speaker’s waveform into frame-level features.
Partition
Sessions
Speakers
Target responses
Duration [h]
Train
195
170
98,329
69.09
Valid
14
10
8,094
4.52
Test
38
31
18,103
12.19
Table 1: Total duration and numbers of sessions, unique speakers, and target responses in each partition.
Type
Chance rate
Proposed
LLM
Main
0.54
4.52
1.62
BC
8.39
19.51
9.55
All
6.41
15.72
7.55
Table 2: Text Top-1 accuracy (%) of short-response selection on 14,446 test events. A selection is correct when its text matches that of the target response. BC denotes backchannel.
Type
1%
5%
10%
25%
50%
Main
7.55
24.32
39.25
62.17
83.98
BC
4.72
18.50
31.55
58.60
83.23
All
5.41
19.97
33.50
59.50
83.42
Table 3: Speech ranking accuracy (%) of the proposed method on 14,446 test events. Top- k % is the rate at which the target speech ranks within the top k % of the candidate pool. The chance rate of Top- k % is k %. BC denotes backchannel.
Candidate type
Min
Max
μ±σ
Speech
29
985
365.71±262.09
Text
23
488
182.63±106.90
Table 4: Number of response candidates per test event ( N=14,446 ).
ϵ (s)
Metric
Method
0.2
0.5
1.0
MAE (s)
Precision
Timing only
60.81
78.14
88.68
-
Proposed
67.04
82.45
91.15
-
Recall
Timing only
54.83
78.48
90.93
-
Proposed
50.35
74.42
88.31
-
F1
Timing only
57.66
78.31
89.79
0.488
Table 5: Event-wise response-onset timing scores (%) and mean absolute error (MAE). ϵ is the absolute timing tolerance in seconds.
Method
Main
BC
Macro
Timing only
79.95
85.73
82.84
Proposed
79.61
86.20
82.90
Table 6: Event-classification F1 scores (%). BC is the event of backchannels.
Case
μ±σ (ms)
P95 (ms)
Proposed
9.3749±0.0672
9.4883
+Select. ( N=101 )
0.0231±0.0007
0.0246
+Select. ( N=102 )
0.0281±0.0008
0.0292
+Select. ( N=103 )
0.0332±0.0009
0.0347
+Select. ( N=104 )
0.0765±0.0009
0.0785
Preface-gen. [ 26 ]
4.3265±0.1175
4.6139
Table 7: Processing times of each method. The upper and lower sections show measurements on the CPU and Metal Performance Shaders (MPS), respectively. N denotes the number of selection candidates. The proposed method operates at 10 Hz, and Nemotron ASR operates at 3.125 Hz. The preface-generation result covers only the response readiness estimation module. Note that subsequent LLM and TTS processing adds further latency in practice.
Large language model (LLM)-based dialogue systems suffer response delays because generation begins only after final speech recognition. While fixed fillers are a workaround, they become unnatural over time. We propose a two-stage incremental framework that decouples prefatory-response preparation from speech onset. Once user intent becomes predictable, an intent readiness detector triggers LLM-based generation of a short prefatory response. Concurrently, a voice activity projection (VAP) model determines when to deliver it. Through a field experiment with a route-guidance robot in a shopping mall, we evaluated three conditions: no-filler, fixed-filler, and contextual-preface. Both fixed-filler and contextual-preface significantly reduced initial response latency relative to no-filler. Relative to fixed-filler, contextual-preface had significantly longer initial response latency but a significantly shorter initial-to-main gap. Exploratory ratings showed no significant differences. These results indicate a timing trade-off.
Yuki Okafuji, Koji Inoue, Yoshiki Ohira
CyberAgent Tokyo, Japan · The University of Osaka Osaka, Japan · Graduate School of Informatics Kyoto University Kyoto, Japan
Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.
Siyuan Liu, Jiahui Xu, Feng Jiang +6
School of Computer Science and Technology, Tongji University · School of Artificial Intelligence, Chinese University of Hong Kong, Shenzhen · Department of Language Science and Technology, Hong Kong Polytechnic University +2
Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path - a duplex S2S model - speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency response onset, while continuing to monitor live audio events. The slow path - a cascaded ASR -> LLM pipeline - generates a higher-quality continuation conditioned on the committed prefix, producing an uninterrupted utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the cascaded pipeline. With GPT-4.1 as the back-end, RelayS2S substantially reduces response latency while preserving nearly all of the cascaded pipeline's textual quality. On synthetic voice dialogues, it achieves a P90 first-chunk latency of 81 ms, excluding TTS and network latency, compared with 1,006 ms for the cascaded baseline. On real voice dialogues, RelayS2S reduces average first-chunk latency by 479 ms while retaining 99% of the cascaded pipeline's textual quality. These benefits become larger as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code is publicly available at: https://github.com/mailong25/relays2s
Long Mai, Junli Liang
Trinity College Dublin, Ireland · University College Dublin, Ireland