Authors: Beomsoo Kim, Changho Choi, Dohyun Kim, Dongki Lee, Ethan Ewer, Eunchong Kim, Gyeongman Kim, Haechan Kim, Hyeonghwan Kim, Inkyu Park, Jihun Yun, Jihwan Moon, Jiyun Kim, Joonghyun Bae, Junhyuck Kim, Minkyu Kim, Sehun Lee, Seungjun Chung, Sungwoo Cho, Dongmin Park, Dongwon Kim, Hara Kang, Jonghyun Lee, Keon Lee, Kangwook Lee, Jaewoong Cho
We present Raon-Speech, a top-performing 9B-parameter speech language model (SpeechLM) for English and Korean speech understanding, answering, and generation, and Raon-SpeechChat, a high-performing full-duplex extension for natural real-time conversation. Raon-Speech successfully transforms a pre-trained LLM into a SpeechLM that both understands and generates speech while preserving strong text capabilities. It trains on 1.38M hours of highly curated English and Korean speech and text datasets with the following training stages: (1) speech modules alignment, (2) end-to-end SpeechLM pre-training with knowledge distillation, and (3) multi-task preference optimization-based post-training. Across 42 English and Korean speech and text benchmarks, Raon-Speech establishes the strongest overall profile on speech-centric tasks in our comparison against eight similarly sized recent audio foundation models, including Qwen2.5-Omni and Fun-Audio-Chat, while preserving strong text question answering performance. Building upon it, Raon-SpeechChat enables natural full-duplex conversation by continual training on 119K hours of time-aligned real and synthetic dialogue data. It proceeds through three complementary training stages: (1) causal encoder adaptation, (2) full-duplex pre-training, (3) full-duplex fine-tuning for voice and role-control. On multiple full-duplex benchmarks, Raon-SpeechChat shows its clearest strengths on the turn-taking and interruption-sensitive behaviors covered by FDB v1.0, and remains competitive across the broader full-duplex evaluation suite. We open-source all model checkpoints, the training and inference pipeline, and an interactive demo.
Figures & tables
Figure 1: Overview of Raon-Speech and Raon-SpeechChat. Raon-Speech extends the backbone LLM to support speech understanding and generation. On the speech understanding side, input speech is passed through the speech encoder and adaptor to obtain speech embeddings, which are then fed into the LLM as input representations. On the speech generation side, the codec embedding from the previous step is first mapped through the output adaptor into the LLM input space. The speech generation expert then predicts the semantic token from the LLM hidden states, after which the residual code predictor (RCP) predicts the acoustic tokens residual-wise for 15 steps. Dequantizing the predicted code at each depth yields codec hidden states, and summing the semantic and all acoustic depths produces the codec embedding. This embedding is passed through the codec decoder to synthesize speech, and is also used as the input to the output adaptor for the next generation step.
Figure 2: Raon-SpeechChat overview. Raon-SpeechChat uses a causal speech encoder to obtain streaming user-speech embeddings and is trained on an interleaved token sequence of user speech, assistant text, and assistant speech. During duplex generation, it continuously listens while speaking in parallel. Dashed vertical lines indicate aligned time boundaries. In the example, the assistant remains silent while listening ( SIL ), then begins speaking with BOW . The text token is generated first; since speech generation lasts longer, PAD tokens are emitted until the spoken word is completed, after which another BOW marks the next word onset.
Task
Input → Output
Simplified rendered example
STT
speech → text
User: <audio> Transcribe the audio into text. Assistant: First, let’s speak about our possible futures and how those are shaped by many agents of change.
TTS
text → speech
User: Speak the following text: There are pulls of the future caused by agents of change, such as social, technological, environmental, economic, and political. Assistant: <audio>
SpeechQA
speech context + text question → text answer
User: <audio> User: How old is the speaker? Assistant: 16
SpokenQA
spoken context + spoken question → text answer
User: <audio> Assistant: Dry palms will produce the most heat when rubbed together because the lack of moisture or lubrication increases friction between the surfaces.
TextQA
text context + text question → text answer
User: What kind of company is Krafton? Assistant: Krafton is a game development and publishing company based in South Korea.
Table 1: Representative pretraining task formats used in Raon-Speech, shown with simplified chat-template-rendered examples. The tasks include STT (transcription), TTS (speech synthesis), SpeechQA and SpokenQA (speech-based question answering), and TextQA (text-based question answering). Audio spans are represented as <audio> for readability.
Stage
Main data / tasks
Objective
Optimization
Training steps
Batch / length
Raon-Speech
Understanding alignment
STT, SpeechQA, SpokenQA
CE
LR 1.5×10−4→1.5×10−5 ; cosine; 950-step warmup
13.5k
batch 128; len 24,576
Generation alignment
TTS
CE
LR 1×10−4→1×10−5 ; cosine; 6,550-step warmup
65.5k
batch 64; len 8,192
Pre-training with KD
All tasks
CE + KL; (on-policy KD)
LR 1.2×10−5→1×10−7 ; cosine; 4,000-step warmup
60k
batch 112; len 8,192
Post-training
Curated SFT (all tasks)
CE + SimPO
LR 1×10−6 ; constant; 112-step warmup
800
batch 96; len 8,192
Raon-SpeechChat
Table 2: Stage-wise training setup of Raon-Speech and Raon-SpeechChat. The table summarizes the main data/tasks and key optimization settings for each stage. Raon-Speech consists of understanding alignment, generation alignment, pre-training with KD, and post-training, while Raon-SpeechChat consists of causal alignment, causal full training, full-duplex training, and two stages of full-duplex fine-tuning.
Figure 3: Overview of data curation. (Left) data pre-processing pipeline for SpeechLMs. (Right) data pre-processing pipeline for full-duplex models.
Raon-Speech
Raon-SpeechChat
Raw Data
Pre-training
Post-training
Pre-training
Fine-tuning
Audio Hours
1.38M
1.01M
104.07K
105.69K
13.85K
# Tokens
124.36B
69.98B
11.19B
10.85B
1.43B
# Samples
614.98M
404.76M
23.36M
4.38M
595.57K
Table 3: Data statistics for Raon-Speech and Raon-SpeechChat across training stages.
Benchmark
Raon
Qwen2.5
Kimi
Audio
Step-Audio
Interactive
Fun-Audio
HyperCLOVA
MiniCPM
-Speech
-Omni
-Audio
Flamingo 3
2 mini
Omni
Chat
X 8B Omni
-o 4.5
Automatic Speech Recognition (WER ↓ )
LibriSpeech-c
1.44
1.73
1.38
1.40
4.88
2.28
1.60
2.28
1.51
LibriSpeech-o
2.89
3.88
2.70
2.97
6.82
4.67
3.89
5.03
3.56
Fleurs-en
3.59
4.05
4.54
4.54
13.02
4.89
7.61
5.57
3.52
Speech Generation (WER ↓ | UTMOS ↑ )
Table 4: English speech and text benchmark results for Raon-Speech. Bold and underline indicate the best and the second-best performance, respectively.
Benchmark
Raon
Qwen2.5
Audio
Step-Audio
Interactive
Fun-Audio
HyperCLOVA
MiniCPM
-Speech
-Omni
Flamingo 3
2 mini
Omni
Chat
X 8B Omni
-o 4.5
Automatic Speech Recognition (CER ↓ )
KSponSpeech-c
6.56
18.96
134.12
55.84
461.87
646.25
10.22
205.35
KSponSpeech-o
6.96
22.72
136.50
59.43
428.83
514.82
10.15
202.14
Fleurs-ko
1.81
3.24
71.85
45.72
159.10
36.44
3.70
168.14
Speech Generation (CER ↓ | UTMOS ↑ )
Table 5: Korean speech and text benchmark results for Raon-Speech. Bold and underline indicate the best and the second-best performance, respectively.
Benchmark Slice
Models
Scenario / Task
Metric
Raon- SpeechChat
Moshi
Freeze- Omni
Persona Plex
MiniCPM-o 4.5
FDB v1.0
Pause Handling
Synthetic TOR ( ↓ )
0.212
0.299
0.620
0.212
0.182
Candor TOR ( ↓ )
0.213
0.370
0.435
0.204
0.343
Backchannel
TOR ( ↓ )
0.091
0.309
0.564
0.236
0.418
Freq. ( ↑ )
0.081
0.050
0.002
0.046
0.008
Table 6: Full-Duplex-Bench summary for Raon-SpeechChat across FDB v1.0, v1.5, and v2.0. Bold and underline indicate the best and the second-best performance, respectively.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Overall performance comparison of Raon-Speech against baseline models on Korean speech and text benchmarks spanning automatic speech recognition (ASR), speech generation, spoken question answering (SpokenQA), speech understanding, and text question answering (TextQA). All scores are zero to max normalized per benchmark axis.
Module
Raon-Speech
Raon-SpeechChat
Shared modules
LLM Backbone (incl. LM Head)
8.2B
Speech Generation Expert (incl. Audio LM Head)
205M
LLM-to-Speech-Generation-Expert Projector
38M
RCP (incl. input projector)
146M
Speech Codec
96M
Appendix
Table 7: Module-wise parameter counts of Raon-Speech and Raon-SpeechChat. Modules above one billion parameters are reported in billions (B), and smaller modules are reported in rounded millions (M). Speaker conditioning includes both the runtime-loaded ECAPA-TDNN backbone and the learned projection.
Pretrained frozen ECAPA-TDNN ( Dawalatabad et al., 2021 ) , followed by a linear projection, 192 → 4096
Appendix
Table 8: Detailed architectural comparison of Raon-Speech and Raon-SpeechChat, with shared modules listed once and speech encoder-related components compared separately. SWA denotes sliding-window attention.
Benchmark
Raon
Qwen2.5
Kimi
Audio
Step-Audio
Interactive
Fun-Audio
HyperCLOVA
MiniCPM
-Speech
-Omni
-Audio
Flamingo3
2 mini
Omni
Chat
X 8B Omni
-o 4.5
VoiceBench ↑
OpenBookQA
86.15
83.30
83.96
62.42
77.14
76.48
81.76
29.01
87.69
MMSU
67.50
56.31
59.82
48.60
54.49
58.72
65.29
30.12
66.69
BBH
85.10
52.90
61.80
29.20
45.90
39.70
67.70
37.90
55.00
SD-QA
60.94
54.07
59.49
38.16
36.35
43.58
61.66
42.68
68.35
Appendix
Table 9: Detailed English spoken question answering results for Raon-Speech. The upper block unpacks the aggregate VoiceBench row and the lower block unpacks the aggregate OpenAudioBench row from the main table. Bold and underline indicate the best and the second-best performance, respectively.
Benchmark
Raon
Qwen2.5
Audio
Step-Audio
Interactive
Fun-Audio
HyperCLOVA
MiniCPM
-Speech
-Omni
Flamingo3
2 mini
Omni
Chat
X 8B Omni
-o 4.5
KVoiceBench ↑
KOpenBookQA
74.83
31.01
6.52
7.42
7.42
28.31
27.42
25.17
KMMSU
55.46
28.34
10.01
11.68
7.02
28.68
23.95
26.85
KBBH
83.47
49.87
36.27
40.27
8.27
52.27
52.67
59.07
KSD-QA
44.84
27.95
7.50
23.26
5.63
32.08
29.08
24.77
Appendix
Table 10: Detailed Korean spoken question answering results for Raon-Speech. The upper block unpacks the aggregate KVoiceBench row and the lower block unpacks the aggregate KOpenAudioBench row from the main table. Bold and underline indicate the best and the second-best performance, respectively.
Capability
Raon- Speech
Qwen2.5- Omni
Audio Flamingo 3
Step-Audio 2 mini
Interactive Omni
Fun-Audio- Chat
HyperCLOVA X 8B Omni
MiniCPM-o 4.5
# Speakers
30.00
29.00
26.00
28.00
23.00
33.00
22.00
36.00
Age
61.96
41.30
21.01
52.17
28.99
47.46
18.00
51.81
Gender
93.70
98.89
57.04
88.89
46.30
87.78
30.00
91.11
Fact Extraction
86.87
81.82
59.60
79.80
33.33
91.92
34.34
88.89
General Counting
65.38
38.46
34.62
40.38
25.00
69.23
26.92
42.31
Topic Summary
92.00
82.00
72.00
91.00
32.00
96.00
70.00
98.00
Appendix
Table 11: Korean speech understanding capability accuracies for KMMAU. Bold and underline indicate the best and the second-best performance, respectively.
Scenario
Model
Respond
Resume
Uncertain
Unknown
User Backchannel
Moshi
0.010
0.092
0.000
0.898
Freeze-Omni
0.010
0.480
0.020
0.490
PersonaPlex
0.020
0.418
0.041
0.520
MiniCPM-o 4.5
0.000
0.520
0.000
0.480
Raon-SpeechChat
0.010
0.398
0.010
0.582
Background Speech
Moshi
0.210
0.100
0.030
0.660
Appendix
Table 12: Scenario-wise FDB v1.5 behavior distributions. Bold and underline indicate the best and the second-best performance, respectively.
Scenario
Model
Stop
Stop n/N
Resp.
Resp. n/N
User Backchannel
Moshi
2.797
12/98
1.895
11/98
Freeze-Omni
4.106
86/98
1.600
38/98
PersonaPlex
3.867
61/98
1.307
38/98
MiniCPM-o 4.5
4.130
91/98
0.154
7/98
Raon-SpeechChat
4.648
81/98
1.897
22/98
Background Speech
Moshi
2.199
15/100
0.858
17/100
Appendix
Table 13: Scenario-wise FDB v1.5 latency details. Bold and underline indicate the best and second-best values, respectively, for the preferred Stop and Response latencies in each scenario.
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.
Qingkai Fang, Shoutao Guo, Yang Feng
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.
Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: https://royalzhang.com/project/lws-page/.