What Matters in Preference Post-Training of Continuous Autoregressive Speech Language Models

Yingda Shen; Yao Qian; Dekun Chen; Yufei Liu; Junan Zhang; Chong Jing; Han Wang; Yuxiang Wang

cs.SDOct 12, 2026

What Matters in Preference Post-Training of Continuous Autoregressive Speech Language Models

Authors: Yingda Shen, Yao Qian, Dekun Chen, Yufei Liu, Junan Zhang, Chong Jing, Han Wang, Yuxiang Wang,

Organizations: The Chinese University of Hong Kong, Shenzhen · Microsoft Research

Abstract

Continuous autoregressive speech language models combine an autoregressive (AR) backbone with a diffusion or flow-matching head. Preference post-training requires choosing how to construct pairs and which modules to update. Controlled interventions show that a perturbed acoustic patch changes subsequent backbone states, yet larger representation changes do not necessarily reduce speech quality. We construct preference pairs through noise or inference-configuration perturbations and formulate a preference update based on reference-relative flow-matching errors. Updating the backbone with a fixed head yields stronger reward improvements than head-only adaptation in our evaluated setting. Across flow-time and module ablations, the lowest training loss does not consistently produce the best speech. These findings support preference post-training that considers both acoustic sampling and autoregressive conditioning, with training choices validated on generated utterances.

Figures & tables

Figure 1: Continuous autoregressive speech language model. The diffusion head generates acoustic patches conditioned on the AR backbone; generated patches become subsequent speech history.

Figure 2: Single-patch intervention at lag 0. RMSE is relative to the anchor; lines average over intervention positions and bands show their range.

Task	Setting	AR $L_{2}$	Latent $L_{2}$	$\Delta$ OVRL
TTS	frozen AR	$0$	$0.95$	$-0.282$
TTS	closed loop	$0.65$	$1.22$	$+0.013$
DNS	frozen AR	$0$	$0.70$	$-0.050$
DNS	closed loop	$0.29$	$0.86$	$+0.022$

Table 1: Noise perturbation with fixed or evolving conditions. Relative

L_{2}

is measured against the anchor;

\Delta

OVRL is the change in DNSMOS overall quality.

Figure 3: Preference-pool analysis: (a) best fixed configuration versus per-prompt oracle reward; (b) comparisons retained by reward-margin filtering.

Figure 4: Training curves for Table 4.2 : (a) flow-time sampling (AR-8); (b) trainable scope.

$t$	$\Delta$ rew.	95% CI	WER	SIM	DNSMOS
Fixed $0.3$	$+0.131$	$[-0.072,0.354]$	$0.124$	$0.454$	$2.85$
Fixed $0.45$	$+0.137$	$[0.025,0.280]$	$0.131$	$0.468$	$2.82$
$[0.3,0.6]$	$+0.069$	$[-0.089,0.253]$	$0.137$	$0.473$	$2.84$
$U(0,1)$	$-0.069$	$[-0.224,0.079]$	$0.145$	$0.468$	$2.80$
Scope	$\Delta$ rew.	95% CI	WER	SIM	DNSMOS
AR last-8	$+0.470$	$[0.181,0.807]$	$0.089$	$0.488$	$2.87$

Table 3: Held-out time and module ablations (95% CI); WER is a fraction. Time: step 300;

[0.3,0.6]

denotes continuous uniform sampling. Scope: best evaluated checkpoint.

	Zero-shot / en				Emotion / en
System	WER $\downarrow$	$\mathrm{SIM_{E}}\uparrow$	$\mathrm{SIM_{W}}\uparrow$	OVRL $\uparrow$	WER $\downarrow$	$\mathrm{SIM_{E}}\uparrow$	$\mathrm{SIM_{W}}\uparrow$	OVRL $\uparrow$	$\mathrm{SIM_{emo}}\uparrow$
CosyVoice3-0.5B	5.44	74.0	0.675	3.12	5.04	66.8	0.576	3.15	0.631
CosyVoice3-0.5B-RL	3.74	73.7	0.657	3.15	5.74	65.7	0.569	3.14	0.591
dots.tts base	5.22	76.1	0.732	3.06	7.17	69.3	0.666	2.90	0.641
dots.tts ARDM-DPO (AR-16)	5.33	75.9	0.732	3.06	7.01	69.6	0.663	2.93	0.576
dots.tts chosen-only SFT	8.38	72.4	0.698	2.98	7.88	67.4	0.643	2.82	0.568

Table 2: Official CV3-Eval English results (WER in percent).

\mathrm{SIM_{E}}

: ERes2Net (percent);

\mathrm{SIM_{W}}

: WavLM cosine;

\mathrm{SIM_{emo}}

: emotion2vec score-distribution cosine. NP/CP: noise/configuration perturbation.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training
Apr 16, 2026Yifu Chen, Shengpeng Ji, Qian Chen +9Spoken Dialogue Systems Multi-Objective Language Model Alignment
Zhejiang University · Tongyi Fun Team, Alibaba Group · Beijing University of Technology
Post-Training Speech Enhancement Language Models with Perceptual Rewards
Jun 19, 2026Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis +1Multi-Objective Reinforcement Learning RL for Language Models
ETH Zürich, Switzerland
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Sep 10, 2026Jingbin Hu, Qirui Zhan, Yuang Cao +7Nonverbal Vocalization Generation Direct Preference Optimization
Shenzhen Pimei Technology Co., Ltd., Guangdong, China · Yutu Zhineng, Beijing, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China +1

Apr 16, 2026J/K move · Enter open · S save

WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training

Yifu Chen, Shengpeng Ji, Qian Chen, Tianle Liang+8

Zhejiang University · Tongyi Fun Team, Alibaba Group · Beijing University of Technology

Spoken Dialogue Systems Multi-Objective Language Model Alignment Speech Language Models

Open paper AskPDF arXiv

End-to-end spoken dialogue models have garnered significant attention because they offer a higher potential ceiling in expressiveness and perceptual ability than cascaded systems. However, the intelligence and expressiveness of current open-source spoken dialogue models often remain below expectations. Motivated by the success of online reinforcement learning(RL) in other domains, one might attempt to directly apply preference optimization to spoken dialogue models, yet this transfer is non-trivial. We analyze these obstacles from the perspectives of reward modeling and rollout sampling, focusing on how sparse preference supervision interacts with dense speech generation under shared-parameter updates. Based on the analysis, we propose a modality-aware adaptive post-training recipe that makes RL practical for spoken dialogue: it constrains preference updates to the semantic channel and improves acoustic behavior via explicit anchoring, while dynamically regulating their mixture from rollout statistics to avoid unreliable preference gradients. We evaluate the method across multiple spoken dialogue benchmarks and representative architectures, and observe consistent improvements in semantic quality and speech expressiveness.

Abstract

Speech Language Models

Explore similar work

WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training

Post-Training Speech Enhancement Language Models with Perceptual Rewards

Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation