Real-Time Short-Response Selection for Spoken Dialogue Systems
Authors: Hien Ohnaka, Kazuyo Onishi, Koichiro Yoshino
Organizations: Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama-cho, Toyonaka, Osaka, 560-8531, Japan · Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, Nara, 630-0101, Japan
To realize spoken dialogue systems that can interact naturally with users, it is important for the system to respond to user utterances at appropriate timings. To achieve such natural turn-taking, the system needs not only to predict appropriate response timings but also to mask delays caused by language generation and speech synthesis so that they are not perceived by the user. We propose a real-time short-response selection task that can be integrated with existing turn-taking models to mask generation delay. To address this task, the proposed model learns the mapping between dialogue context and short responses using contrastive learning. When a turn-taking event is detected, the system selects a response from a set of pre-generated candidates. The selected response is played while the main response is being generated. Experimental results show that the proposed method learns to select short responses based on dialogue context, outperforming random and LLM-based baselines. Moreover, integrating response selection maintained turn-taking prediction performance and added minimal processing overhead.
Figures & tables
Figure 1: Overview of the proposed real-time short-response selection.
Figure 2: Original VAP architecture used to initialize the turn-taking backbone. The contrastive predictive coding (CPC) encoder [ 32 ] is used as the feature extractor that converts each speaker’s waveform into frame-level features.
Partition
Sessions
Speakers
Target responses
Duration [h]
Train
195
170
98,329
69.09
Valid
14
10
8,094
4.52
Test
38
31
18,103
12.19
Table 1: Total duration and numbers of sessions, unique speakers, and target responses in each partition.
Type
Chance rate
Proposed
LLM
Main
0.54
4.52
1.62
BC
8.39
19.51
9.55
All
6.41
15.72
7.55
Table 2: Text Top-1 accuracy (%) of short-response selection on 14,446 test events. A selection is correct when its text matches that of the target response. BC denotes backchannel.
Type
1%
5%
10%
25%
50%
Main
7.55
24.32
39.25
62.17
83.98
BC
4.72
18.50
31.55
58.60
83.23
All
5.41
19.97
33.50
59.50
83.42
Table 3: Speech ranking accuracy (%) of the proposed method on 14,446 test events. Top- k % is the rate at which the target speech ranks within the top k % of the candidate pool. The chance rate of Top- k % is k %. BC denotes backchannel.
Candidate type
Min
Max
μ±σ
Speech
29
985
365.71±262.09
Text
23
488
182.63±106.90
Table 4: Number of response candidates per test event ( N=14,446 ).
ϵ (s)
Metric
Method
0.2
0.5
1.0
MAE (s)
Precision
Timing only
60.81
78.14
88.68
-
Proposed
67.04
82.45
91.15
-
Recall
Timing only
54.83
78.48
90.93
-
Proposed
50.35
74.42
88.31
-
F1
Timing only
57.66
78.31
89.79
0.488
Table 5: Event-wise response-onset timing scores (%) and mean absolute error (MAE). ϵ is the absolute timing tolerance in seconds.
Method
Main
BC
Macro
Timing only
79.95
85.73
82.84
Proposed
79.61
86.20
82.90
Table 6: Event-classification F1 scores (%). BC is the event of backchannels.
Case
μ±σ (ms)
P95 (ms)
Proposed
9.3749±0.0672
9.4883
+Select. ( N=101 )
0.0231±0.0007
0.0246
+Select. ( N=102 )
0.0281±0.0008
0.0292
+Select. ( N=103 )
0.0332±0.0009
0.0347
+Select. ( N=104 )
0.0765±0.0009
0.0785
Preface-gen. [ 26 ]
4.3265±0.1175
4.6139
Table 7: Processing times of each method. The upper and lower sections show measurements on the CPU and Metal Performance Shaders (MPS), respectively. N denotes the number of selection candidates. The proposed method operates at 10 Hz, and Nemotron ASR operates at 3.125 Hz. The preface-generation result covers only the response readiness estimation module. Note that subsequent LLM and TTS processing adds further latency in practice.
School of Computer Science and Technology, Tongji University · School of Artificial Intelligence, Chinese University of Hong Kong, Shenzhen · Department of Language Science and Technology, Hong Kong Polytechnic University +2