Organizations: MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai AI Laboratory · MOSI Intelligence · SenseTime Group Inc. · Shenzhen University of Advanced Technology
General turn-taking behavior in real-time dialogue systems requires deciding whether to keep listening or start responding while listening, and whether to continue or stop while speaking. Existing turn detectors use heterogeneous, task-specific label spaces and are often trained on limited annotations or evaluated on isolated utterances, making them difficult to use as a unified causal controller with comprehensive context. We propose XTurnix, a compact text-based model that formulates turn control as two binary decisions conditioned on the AI's current listening or speaking state and predicts a single control token from the complete dialogue history. XTurnix is pretrained on 5.5 million causal action examples automatically derived from timestamped two-speaker transcripts, then fine-tuned on synthetic multi-turn examples with a flatter distribution across the four state-action labels. We evaluate XTurnix on four public benchmarks and a balanced self-curated benchmark. Across the public benchmarks, XTurnix achieves the best results on all SemanticVAD and LiveKit splits, ties the native Smart-Turn model on Smart-Turn Bench, and achieves the highest incomplete-turn accuracy on Easy-Turn. On the self-curated benchmark, it reaches 89.06% accuracy, more than 20 percentage points above the strongest third-party baseline at 68.75%, while maintaining F1 scores between 84.21% and 90.63% across all four categories. These results demonstrate unified listening- and speaking-state turn control in a single compact model. Code is available at https://github.com/xcc-zach/xturnix, with an interactive demo at https://huggingface.co/spaces/xcczach/xturnix-demo.
Figures & tables
Figure 1: XTurnix overview
Current AI state
Interpretation of the latest user input
AI Action
Next AI state
Listening
The user has not yielded the current turn.
<|keep|>
Listening
Listening
The user has yielded the turn and a response may begin.
<|start|>
Speaking
Speaking
The input is a backchannel or overlap that does not request the floor.
<|keep|>
Speaking
Speaking
The input is an interruption that requests the floor.
<|stop|>
Listening
Table 1: The two-state binary turn-detection formulation.
Figure 2: XTurnix training stages
State
Action
Earlier dialogue
Latest user input
Listening
<|keep|>
User: I am considering a lakeside town for my weekend trip. AI: Why does that destination appeal to you?
User: Because it
Listening
<|start|>
User: I am considering a lakeside town for my weekend trip. AI: Why does that destination appeal to you?
User: Because it is quiet
Speaking
<|keep|>
User: How does prefill differ from decoding? AI: Prefill processes the prompt in parallel, whereas decoding generates one token at a time and uses
User: Mm-hm
Speaking
<|stop|>
User: I loaded the smaller checkpoint. AI: So it is the 1.7B version, right? If not
User: Yes
Table 2: Examples of the four training classes formed by the two current states and their two legal outputs. The latest user input is the decision point; incomplete text intentionally has no terminal punctuation.
Easy-Turn Bench
Smart-Turn Bench
SemanticVAD Bench
LiveKit Bench
Model
ZH
All
ZH
EN
All
ZH
EN
All
ZH
EN
Easy-Turn
850M
97.33
–
13.31
–
–
70.06
–
–
55.45
–
Smart-Turn
8M
77.17
98.81
98.52
99.00
54.61
53.09
69.75
70.91
64.60
76.24
FireRedChat
170M
81.17
9.29
1.78
14.34
72.42
72.59
70.75
42.67
42.91
42.48
Namo
307M
50.83
44.17
48.52
41.24
46.14
45.68
50.75
49.92
44.20
54.75
TEN
7B
90.33
74.40
74.85
74.10
62.65
61.14
77.75
63.50
61.20
65.45
Table 3: Accuracy (%) on Chinese (ZH) and English (EN). Easy-Turn Bench excludes backchannel examples, and Smart-Turn Bench is restricted to endfiller=true . Best results are bold, and second-best underlined.
Table 6
Evaluation slice
Model
ROC-AUC
<|start|> precision
<|start|> recall
<|keep|> recall
Easy-Turn endpoint
Qwen3-0.6B
35.34
49.75
99.00
0.00
XTurnix-PT
99.28
100.00
69.00
100.00
XTurnix-ZH-Base
99.28
99.28
91.33
99.33
Smart-Turn, endfiller=false
Qwen3-0.6B
57.88
80.03
100.00
0.00
XTurnix-PT
87.92
94.74
15.22
96.61
XTurnix-ZH-Base
87.36
94.92
51.37
88.98
Table 6: Turn-completion results (%) on Easy-Turn endpoint examples and Smart-Turn (ZH) examples with endfiller=false . ROC-AUC measures how well the model separates <|start|> from <|keep|> across all thresholds. Easy-Turn results exclude backchannels.
Overall
Listening F1
Speaking F1
Model
Accuracy
<|keep|>
<|start|>
<|keep|>
<|stop|>
Easy-Turn
57.03
65.31
78.48
38.71
42.42
Smart-Turn
47.66
61.22
75.95
11.11
35.14
FireRedChat
61.72
0.00
66.67
69.09
76.71
Namo
46.88
36.36
52.05
44.07
52.17
TEN
51.56
93.55
93.94
3.33
14.71
Table 7: Overall accuracy and class-specific F1 scores (%) on the self-curated benchmark. Best results in each column are bold, and second-best results underlined.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
All
EN
Model
False
True
Missing
False
True
Missing
Smart-Turn
87.80
98.81
95.35
88.82
99.00
95.35
FireRedChat
75.24
9.29
51.19
72.23
14.34
51.19
Namo
54.45
44.17
53.95
54.80
41.24
53.95
TEN
82.81
74.40
79.08
81.92
74.10
79.08
TurnSense
53.97
79.40
44.69
49.11
68.92
44.69
Appendix
Table 8: Pooled (All) and English (EN) accuracy (%) on Smart-Turn Bench grouped by endfiller , extending Table 5 . False includes only explicitly false annotations; missing metadata is shown separately. Best results in each column are bold, and second-best results underlined.
All
EN
Model
ROC-AUC
Start P
Start R
Keep R
ROC-AUC
Start P
Start R
Keep R
Qwen3-0.6B
53.36
75.12
100.00
0.00
56.50
72.41
100.00
0.00
XTurnix-PT
87.29
94.29
7.92
98.55
86.44
93.10
3.47
99.32
XTurnix-ZH-Base
87.00
94.56
32.00
94.44
86.33
94.01
20.21
96.62
Appendix
Table 9: Pooled (All) and English (EN) turn-completion metrics (%) on Smart-Turn samples explicitly annotated as endfiller=false , extending Table 6 .
Accurate and responsive turn-taking is essential for spoken dialogue systems, which must distinguish in real time between user interruptions, backchannels that should be ignored, and the completion of an utterance. Prior modular approaches typically optimize turn state prediction at the utterance or fixed-chunk level, creating a mismatch with the continuous turn state estimate, and often depend on an auxiliary ASR model, which limits responsiveness and increases overall system complexity. Therefore, we present X2-Turn, a frame-synchronous turn state prediction method via delayed-stream modeling. Specifically, building on the pretrained Voxtral Realtime model, we introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations, jointly predicting ASR tokens and fine-grained turn states at the frame level. Experiments on bilingual EasyTurn and Full-Duplex-Bench demonstrate that the proposed method achieves an effective trade-off between turn state accuracy and decision latency.
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.