Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI agents. Existing systems often rely on acoustic or linguistic cues alone, limiting accuracy and stability, whereas directly endowing large language models with full-duplex capability incurs heavy data and deployment costs that hinder real-time usage. We propose JAL-Turn, a lightweight speech-only turn-taking framework based on a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively fuses pre-trained acoustic and linguistic representations for low-latency hold or shift prediction. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing almost no additional end-to-end latency. We further introduce a scalable pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Experiments on the public Easy-Turn and Smart-Turn benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn achieves competing results among strong baselines in detection accuracy while maintaining superior real-time performance.
Figures & tables
Figure 1: Overall training architecture of the proposed JAL-Turn framework.
Figure 2: Schematics of the overall labeling method.
Model
Acc cp
Acc incp
Acc bc
Acc wait
Latency (ms)
Paraformer [ 7 ] + TEN Turn Detection
86.67
89.30
-
91.00
79
STurn-v2
78.67
62.00
-
-
16
ETurn [ 14 ]
96.33
97.67
91.00
98.00
143
JAL-Turn
96.67
93.67
80.00
92.00
12
Table 1: Comparison of turn-taking detection methods on Mandarin Easy-Turn test set. “-” indicates that the state is not applicable to the given baseline.
Model
Smart-Turn (multilingual)
In-house Japanese
Acc
F1
Latency (ms)
Acc
F1
Latency (ms)
STurn-v2
65.12
0.679
48
55.46
0.427
51
STurn-v3
93.10
0.931
12
71.94
0.736
13
JAL-Turn
93.27
0.934
36
93.24
0.939
38
Table 2: Performance comparison of audio-only turn-taking detection methods on the multilingual Smart-Turn benchmark and the in-house Japanese corpus.
Model
Acc
F1
Latency (ms)
Gemini-2.5-Flash
76.91
0.817
595
Qwen3-0.6B
78.70
0.782
124
GPT-5.1
85.52
0.874
1205
JAL-Turn
93.24
0.939
38
Table 3: Performance comparison of JAL-Turn against LLM-based turn-taking detection methods on the in-house benchmark.
Model
Acc
F1
Latency (ms)
JAL-Turn
93.24
0.939
38
w/o Sense
72.01
0.698
12
w/o CPC
84.18
0.839
26
w/o CrossATT
88.59
0.873
40
w/o ATTPooling
90.23
0.895
45
Table 4: Ablation studies of the proposed JAL-Turn. w/o CrossATT denotes using concatenation, w/o ATTPooling represents using the last layer feature.
Figure 3: Encoder-wise contributions across utterance durations for SenseVoice and CPC.
Figure 4: Temporal-level attribution heatmaps for the three languages (normalized to 0 – 100% ). The upper and lower rows correspond to the SenseVoice and CPC encoders, respectively. Red dashed lines indicate the boundary between Hold and Shift samples.
Recent advances in AudioLLMs have enabled spoken dialogue systems to move beyond turn-based interaction toward real-time full-duplex communication, where the agent must decide when to speak, yield, or interrupt while the user is still talking. Existing full-duplex approaches either rely on voice activity cues, which lack semantic understanding, or on ASR-based modules, which introduce latency and degrade under overlapping speech and noise. Moreover, available datasets rarely capture realistic interaction dynamics, limiting evaluation and deployment. To mitigate the problem, we propose FastTurn, a unified framework for low-latency and robust turn detection. To advance latency while maintaining performance, FastTurn combines streaming CTC decoding with acoustic features, enabling early decisions from partial observations while preserving semantic cues. We also release a test set based on real human dialogue, capturing authentic turn transitions, overlapping speech, backchannels, pauses, pitch variation, and environmental noise. Experiments show FastTurn achieves higher decision accuracy with lower interruption latency than representative baselines and remains robust under challenging acoustic conditions, demonstrating its effectiveness for practical full-duplex dialogue systems.
Accurate and responsive turn-taking is essential for spoken dialogue systems, which must distinguish in real time between user interruptions, backchannels that should be ignored, and the completion of an utterance. Prior modular approaches typically optimize turn state prediction at the utterance or fixed-chunk level, creating a mismatch with the continuous turn state estimate, and often depend on an auxiliary ASR model, which limits responsiveness and increases overall system complexity. Therefore, we present X2-Turn, a frame-synchronous turn state prediction method via delayed-stream modeling. Specifically, building on the pretrained Voxtral Realtime model, we introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations, jointly predicting ASR tokens and fine-grained turn states at the frame level. Experiments on bilingual EasyTurn and Full-Duplex-Bench demonstrate that the proposed method achieves an effective trade-off between turn state accuracy and decision latency.
In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of different modalities remains unclear. We present a controlled ablation of acoustic, prosodic, and semantic signals for streaming end of turn detection using a lightweight trimodal classifier. Under identical training conditions, the acoustic prosodic combination achieves the best balance of accuracy and latency, achieving utterance F1 of 0.93 with 7.8% false alarms at 400ms median latency. Adding text increases premature detections without improving performance. Feature space analysis confirms that prosodic features have the strongest class separability, while text representations overlap substantially. These findings suggest that turn-taking is primarily conveyed through intonation and silence patterns rather than semantic completeness, enabling faster and more reliable systems without expensive text inference.