While the shift from cascaded dialogue systems to end-to-end (E2E) speech Large Language Models (LLMs) improves latency and paralinguistic modeling, E2E models often exhibit a significant performance degradation compared to their text-based counterparts. The standard Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) training methods fail to close this gap. To address this, we propose X-OPD, a novel Cross-Modal On-Policy Distillation framework designed to systematically align the capabilities of Speech LLMs to their text-based counterparts. X-OPD enables the Speech LLM to explore its own distribution via on-policy rollouts, where a text-based teacher model evaluates these trajectories and provides token-level feedback, effectively distilling teacher's capabilities into student's multi-modal representations. Extensive experiments across multiple benchmarks demonstrate that X-OPD significantly narrows the gap in complex tasks while preserving the model's inherent capabilities.
Figures & tables
Figure 1: Overview of the proposed Cross-Modal On-Policy Distillation (X-OPD) framework.
Table 1: Performance comparison across different model series and training paradigms. We report performance scores across multiple benchmarks for both speech-input (S) and text-input (T) modalities. The Avg. Drop (%) measures the aggregate performance gap relative to each series’ original Base Model (for Gemini, its own text modality serves as the baseline).
Method
BIG Bench Audio
Audio Multi-Chall.
Voice Bench
Avg. Drop (%)
S
T
S
T
S
T
ΔS
ΔT
Reference (Text-only Models)
Qwen3-A22B-Instruct
–
99.10
–
35.73
–
94.09
–
–
Qwen3-A3B-Instruct
–
97.72
–
29.31
–
91.00
–
–
Qwen3-Omni-A3B-Instruct
85.67
92.10
23.57
26.14
89.22
91.04
11.29
5.51
X-OPD
93.41
96.85
28.14
28.66
89.29
91.18
3.43
0.97
Table 2: Ablation Study of X-OPD. The results are presented in the same format as Table 1 . The default X-OPD configuration is based on the Qwen3-Omni-A3B-Instruct, utilizing Qwen3-A3B-Instruct as the teacher with λ=0.5 .
Method
MMAR (%)
Drop
Original Model
71.3
-
SFT
60.3
11.0
Offline KD
59.9
11.4
GKD (Forward KL)
60.0
11.3
X-OPD ( λ=0 )
69.8
1.5
X-OPD ( λ=0.5 )
69.3
2.0
Table 3: Analysis of catastrophic forgetting on the MMAR benchmark. Total accuracy ( ↑ ) is reported.
Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)
Yan Zhou, Qingkai Fang, Yun Hong +1
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
While large audio-language models have achieved remarkable progress in auditory perception, they still lag behind text-based large language models in deep logical reasoning, primarily due to the scarcity of high-quality audio reasoning data. To bridge this gap, we propose X3-OPD, a cross-modal on-policy distillation framework that transfers reasoning capabilities from a powerful text teacher to an audio-language student. During training, the student generates reasoning trajectories conditioned on its own acoustic perception, while the teacher provides token-level guidance using matched textual inputs and verified answers. We further construct a three-tier symmetric corpus covering textual reasoning rendered into speech, audio-event reasoning grounded in complex acoustic scenes, and spoken-dialogue reasoning involving paralinguistic cues. This design extends cross-modal distillation beyond textually recoverable content to reasoning grounded in non-linguistic events, prosody, and conversational context. Experiments on MMSU, MMAU, BIG Bench Audio, and MMAR demonstrate that X3-OPD substantially improves audio-grounded reasoning and chain-of-thought quality while largely preserving the model's existing capabilities under domain shift.
Speech Language Models (SLMs) inherit strong instruction-following capabilities from pretrained language models, yet ASR specialization can substantially degrade them. To address this ASR--QA trade-off, we propose Task-Specific On-Policy Distillation (TS-OPD), which leverages models before and after ASR specialization as complementary QA and ASR teachers. The student generates separate task-conditioned trajectories for ASR and QA, each supervised only by its corresponding teacher, thereby reducing direct competition between the two supervision signals. Experiments on basic ASR, contextual ASR, and QA demonstrate that TS-OPD improves recognition while preserving QA capability. Moreover, TS-OPD remains robust across different balancing coefficients and continues to benefit from increased distillation data.
Yujie Guo, Hongjie Chen, Jian Kang +3
College of Computer Science, Nankai University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd