Raon-Speech Technical Report

Beomsoo Kim; Changho Choi; Dohyun Kim; Dongki Lee; Ethan Ewer; Eunchong Kim; Gyeongman Kim; Haechan Kim

cs.CLApr 8, 2026

Raon-Speech Technical Report

Authors: Beomsoo Kim, Changho Choi, Dohyun Kim, Dongki Lee, Ethan Ewer, Eunchong Kim, Gyeongman Kim, Haechan Kim,

Organizations: KRAFTON

Abstract

We present Raon-Speech, a top-performing 9B-parameter speech language model (SpeechLM) for English and Korean speech understanding, answering, and generation, and Raon-SpeechChat, a high-performing full-duplex extension for natural real-time conversation. Raon-Speech successfully transforms a pre-trained LLM into a SpeechLM that both understands and generates speech while preserving strong text capabilities. It trains on 1.38M hours of highly curated English and Korean speech and text datasets with the following training stages: (1) speech modules alignment, (2) end-to-end SpeechLM pre-training with knowledge distillation, and (3) multi-task preference optimization-based post-training. Across 42 English and Korean speech and text benchmarks, Raon-Speech establishes the strongest overall profile on speech-centric tasks in our comparison against eight similarly sized recent audio foundation models, including Qwen2.5-Omni and Fun-Audio-Chat, while preserving strong text question answering performance. Building upon it, Raon-SpeechChat enables natural full-duplex conversation by continual training on 119K hours of time-aligned real and synthetic dialogue data. It proceeds through three complementary training stages: (1) causal encoder adaptation, (2) full-duplex pre-training, (3) full-duplex fine-tuning for voice and role-control. On multiple full-duplex benchmarks, Raon-SpeechChat shows its clearest strengths on the turn-taking and interruption-sensitive behaviors covered by FDB v1.0, and remains competitive across the broader full-duplex evaluation suite. We open-source all model checkpoints, the training and inference pipeline, and an interactive demo.

Figures & tables

Figure 1: Overview of Raon-Speech and Raon-SpeechChat. Raon-Speech extends the backbone LLM to support speech understanding and generation. On the speech understanding side, input speech is passed through the speech encoder and adaptor to obtain speech embeddings, which are then fed into the LLM as input representations. On the speech generation side, the codec embedding from the previous step is first mapped through the output adaptor into the LLM input space. The speech generation expert then predicts the semantic token from the LLM hidden states, after which the residual code predictor (RCP) predicts the acoustic tokens residual-wise for 15 steps. Dequantizing the predicted code at each depth yields codec hidden states, and summing the semantic and all acoustic depths produces the codec embedding. This embedding is passed through the codec decoder to synthesize speech, and is also used as the input to the output adaptor for the next generation step.

Figure 2: Raon-SpeechChat overview. Raon-SpeechChat uses a causal speech encoder to obtain streaming user-speech embeddings and is trained on an interleaved token sequence of user speech, assistant text, and assistant speech. During duplex generation, it continuously listens while speaking in parallel. Dashed vertical lines indicate aligned time boundaries. In the example, the assistant remains silent while listening ( SIL ), then begins speaking with BOW . The text token is generated first; since speech generation lasts longer, PAD tokens are emitted until the spoken word is completed, after which another BOW marks the next word onset.

Task	Input $\rightarrow$ Output	Simplified rendered example
STT	speech $\rightarrow$ text	User: <audio> Transcribe the audio into text. Assistant: First, let’s speak about our possible futures and how those are shaped by many agents of change.
TTS	text $\rightarrow$ speech	User: Speak the following text: There are pulls of the future caused by agents of change, such as social, technological, environmental, economic, and political. Assistant: <audio>
SpeechQA	speech context + text question $\rightarrow$ text answer	User: <audio> User: How old is the speaker? Assistant: 16
SpokenQA	spoken context + spoken question $\rightarrow$ text answer	User: <audio> Assistant: Dry palms will produce the most heat when rubbed together because the lack of moisture or lubrication increases friction between the surfaces.
TextQA	text context + text question $\rightarrow$ text answer	User: What kind of company is Krafton? Assistant: Krafton is a game development and publishing company based in South Korea.

Table 1: Representative pretraining task formats used in Raon-Speech, shown with simplified chat-template-rendered examples. The tasks include STT (transcription), TTS (speech synthesis), SpeechQA and SpokenQA (speech-based question answering), and TextQA (text-based question answering). Audio spans are represented as <audio> for readability.

Stage	Main data / tasks	Objective	Optimization	Training steps	Batch / length
Raon-Speech
Understanding alignment	STT, SpeechQA, SpokenQA	CE	LR $1.5\times 10^{-4}\rightarrow 1.5\times 10^{-5}$ ; cosine; 950-step warmup	13.5k	batch 128; len 24,576
Generation alignment	TTS	CE	LR $1\times 10^{-4}\rightarrow 1\times 10^{-5}$ ; cosine; 6,550-step warmup	65.5k	batch 64; len 8,192
Pre-training with KD	All tasks	CE + KL; (on-policy KD)	LR $1.2\times 10^{-5}\rightarrow 1\times 10^{-7}$ ; cosine; 4,000-step warmup	60k	batch 112; len 8,192
Post-training	Curated SFT (all tasks)	CE + SimPO	LR $1\times 10^{-6}$ ; constant; 112-step warmup	800	batch 96; len 8,192
Raon-SpeechChat

Table 2: Stage-wise training setup of Raon-Speech and Raon-SpeechChat. The table summarizes the main data/tasks and key optimization settings for each stage. Raon-Speech consists of understanding alignment, generation alignment, pre-training with KD, and post-training, while Raon-SpeechChat consists of causal alignment, causal full training, full-duplex training, and two stages of full-duplex fine-tuning.

Figure 3: Overview of data curation. (Left) data pre-processing pipeline for SpeechLMs. (Right) data pre-processing pipeline for full-duplex models.

		Raon-Speech		Raon-SpeechChat
	Raw Data	Pre-training	Post-training	Pre-training	Fine-tuning
Audio Hours	1.38M	1.01M	104.07K	105.69K	13.85K
# Tokens	124.36B	69.98B	11.19B	10.85B	1.43B
# Samples	614.98M	404.76M	23.36M	4.38M	595.57K

Table 3: Data statistics for Raon-Speech and Raon-SpeechChat across training stages.

Benchmark	Raon	Qwen2.5	Kimi	Audio	Step-Audio	Interactive	Fun-Audio	HyperCLOVA	MiniCPM
	-Speech	-Omni	-Audio	Flamingo 3	2 mini	Omni	Chat	X 8B Omni	-o 4.5
Automatic Speech Recognition (WER $\downarrow$ )
LibriSpeech-c	1.44	1.73	1.38	1.40	4.88	2.28	1.60	2.28	1.51
LibriSpeech-o	2.89	3.88	2.70	2.97	6.82	4.67	3.89	5.03	3.56
Fleurs-en	3.59	4.05	4.54	4.54	13.02	4.89	7.61	5.57	3.52
Speech Generation (WER $\downarrow$ \| UTMOS $\uparrow$ )

Table 4: English speech and text benchmark results for Raon-Speech. Bold and underline indicate the best and the second-best performance, respectively.

Benchmark	Raon	Qwen2.5	Audio	Step-Audio	Interactive	Fun-Audio	HyperCLOVA	MiniCPM
	-Speech	-Omni	Flamingo 3	2 mini	Omni	Chat	X 8B Omni	-o 4.5
Automatic Speech Recognition (CER $\downarrow$ )
KSponSpeech-c	6.56	18.96	134.12	55.84	461.87	646.25	10.22	205.35
KSponSpeech-o	6.96	22.72	136.50	59.43	428.83	514.82	10.15	202.14
Fleurs-ko	1.81	3.24	71.85	45.72	159.10	36.44	3.70	168.14
Speech Generation (CER $\downarrow$ \| UTMOS $\uparrow$ )

Table 5: Korean speech and text benchmark results for Raon-Speech. Bold and underline indicate the best and the second-best performance, respectively.

Benchmark Slice		Models
Scenario / Task	Metric	Raon- SpeechChat	Moshi	Freeze- Omni	Persona Plex	MiniCPM-o 4.5
FDB v1.0
Pause Handling	Synthetic TOR ( $\downarrow$ )	0.212	0.299	0.620	0.212	0.182
	Candor TOR ( $\downarrow$ )	0.213	0.370	0.435	0.204	0.343
Backchannel	TOR ( $\downarrow$ )	0.091	0.309	0.564	0.236	0.418
	Freq. ( $\uparrow$ )	0.081	0.050	0.002	0.046	0.008

Table 6: Full-Duplex-Bench summary for Raon-SpeechChat across FDB v1.0, v1.5, and v2.0. Bold and underline indicate the best and the second-best performance, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overall performance comparison of Raon-Speech against baseline models on Korean speech and text benchmarks spanning automatic speech recognition (ASR), speech generation, spoken question answering (SpokenQA), speech understanding, and text question answering (TextQA). All scores are zero to max normalized per benchmark axis.

Module	Raon-Speech	Raon-SpeechChat
Shared modules
LLM Backbone (incl. LM Head)	8.2B
Speech Generation Expert (incl. Audio LM Head)	205M
LLM-to-Speech-Generation-Expert Projector	38M
RCP (incl. input projector)	146M
Speech Codec	96M

Appendix

Table 7: Module-wise parameter counts of Raon-Speech and Raon-SpeechChat. Modules above one billion parameters are reported in billions (B), and smaller modules are reported in rounded millions (M). Speaker conditioning includes both the runtime-loaded ECAPA-TDNN backbone and the learned projection.

Component	Raon-Speech	Raon-SpeechChat
Shared modules
LLM Backbone	Hidden 4096, 36 layers, FFN 12288 (SwiGLU), RoPE
Speech Generation Stack	Speech Generation Expert: Hidden 2048, 4 layers, FFN 6144 (SwiGLU), RoPE LLM-to-Expert Projector: 2-layer MLP, 4096 $\rightarrow$ 6144 $\rightarrow$ 2048, SiLU RCP: Hidden 1024, 5 layers
Speech Codec	Mimi conv-transformer autoencoder; hidden 512; codebook 2048; 16/32 RVQ groups 12.5 Hz; causal SWA (10 s window)
Speech Output Adaptor	2-layer MLP, 512 $\rightarrow$ 4096, GELU; post-RMSNorm (init=0.02)
Speaker Conditioning	Pretrained frozen ECAPA-TDNN ( Dawalatabad et al., 2021 ) , followed by a linear projection, 192 $\rightarrow$ 4096

Appendix

Table 8: Detailed architectural comparison of Raon-Speech and Raon-SpeechChat, with shared modules listed once and speech encoder-related components compared separately. SWA denotes sliding-window attention.

Benchmark	Raon	Qwen2.5	Kimi	Audio	Step-Audio	Interactive	Fun-Audio	HyperCLOVA	MiniCPM
	-Speech	-Omni	-Audio	Flamingo3	2 mini	Omni	Chat	X 8B Omni	-o 4.5
VoiceBench $\uparrow$
OpenBookQA	86.15	83.30	83.96	62.42	77.14	76.48	81.76	29.01	87.69
MMSU	67.50	56.31	59.82	48.60	54.49	58.72	65.29	30.12	66.69
BBH	85.10	52.90	61.80	29.20	45.90	39.70	67.70	37.90	55.00
SD-QA	60.94	54.07	59.49	38.16	36.35	43.58	61.66	42.68	68.35

Appendix

Table 9: Detailed English spoken question answering results for Raon-Speech. The upper block unpacks the aggregate VoiceBench row and the lower block unpacks the aggregate OpenAudioBench row from the main table. Bold and underline indicate the best and the second-best performance, respectively.

Benchmark	Raon	Qwen2.5	Audio	Step-Audio	Interactive	Fun-Audio	HyperCLOVA	MiniCPM
	-Speech	-Omni	Flamingo3	2 mini	Omni	Chat	X 8B Omni	-o 4.5
KVoiceBench $\uparrow$
KOpenBookQA	74.83	31.01	6.52	7.42	7.42	28.31	27.42	25.17
KMMSU	55.46	28.34	10.01	11.68	7.02	28.68	23.95	26.85
KBBH	83.47	49.87	36.27	40.27	8.27	52.27	52.67	59.07
KSD-QA	44.84	27.95	7.50	23.26	5.63	32.08	29.08	24.77

Appendix

Table 10: Detailed Korean spoken question answering results for Raon-Speech. The upper block unpacks the aggregate KVoiceBench row and the lower block unpacks the aggregate KOpenAudioBench row from the main table. Bold and underline indicate the best and the second-best performance, respectively.

Capability	Raon- Speech	Qwen2.5- Omni	Audio Flamingo 3	Step-Audio 2 mini	Interactive Omni	Fun-Audio- Chat	HyperCLOVA X 8B Omni	MiniCPM-o 4.5
# Speakers	30.00	29.00	26.00	28.00	23.00	33.00	22.00	36.00
Age	61.96	41.30	21.01	52.17	28.99	47.46	18.00	51.81
Gender	93.70	98.89	57.04	88.89	46.30	87.78	30.00	91.11
Fact Extraction	86.87	81.82	59.60	79.80	33.33	91.92	34.34	88.89
General Counting	65.38	38.46	34.62	40.38	25.00	69.23	26.92	42.31
Topic Summary	92.00	82.00	72.00	91.00	32.00	96.00	70.00	98.00

Appendix

Table 11: Korean speech understanding capability accuracies for KMMAU. Bold and underline indicate the best and the second-best performance, respectively.

Scenario	Model	Respond	Resume	Uncertain	Unknown
User Backchannel	Moshi	0.010	0.092	0.000	0.898
	Freeze-Omni	0.010	0.480	0.020	0.490
	PersonaPlex	0.020	0.418	0.041	0.520
	MiniCPM-o 4.5	0.000	0.520	0.000	0.480
	Raon-SpeechChat	0.010	0.398	0.010	0.582
Background Speech	Moshi	0.210	0.100	0.030	0.660

Appendix

Table 12: Scenario-wise FDB v1.5 behavior distributions. Bold and underline indicate the best and the second-best performance, respectively.

Scenario	Model	Stop	Stop $n/N$	Resp.	Resp. $n/N$
User Backchannel	Moshi	2.797	12/98	1.895	11/98
	Freeze-Omni	4.106	86/98	1.600	38/98
	PersonaPlex	3.867	61/98	1.307	38/98
	MiniCPM-o 4.5	4.130	91/98	0.154	7/98
	Raon-SpeechChat	4.648	81/98	1.897	22/98
Background Speech	Moshi	2.199	15/100	0.858	17/100

Appendix

Table 13: Scenario-wise FDB v1.5 latency details. Bold and underline indicate the best and second-best values, respectively, for the preferred Stop and Response latencies in each scenario.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Sep 18, 2026Jagadeesh Balam, Travis Bartley, Edresson Casanova +52Tool-Augmented Language Model Agents Speech Language Models
Wookie
Liberating LLM Capabilities in Full-Duplex Speech Models
May 4, 2026Luoyuan Zhang, Bokai Xu, Junbo Cui +4Speech Language Models Full-Duplex Spoken Dialogue Systems
1CUHK(SZ) · 2ModelBest · 4Peking University +1

Jun 12, 2026J/K move · Enter open · S save

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Spoken Dialogue Systems Speech Language Models Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue

Open paper AskPDF arXiv

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Abstract

Speech Language Models

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Liberating LLM Capabilities in Full-Duplex Speech Models