ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models

Chi-Yuan Hsiao; Ke-Han Lu; Yu-Kuan Fu; Guan-Ting Lin; Hsiao-Tsung Hung; Hung-yi Lee

cs.CLApr 11, 2026

ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models

Authors: Chi-Yuan Hsiao, Ke-Han Lu, Yu-Kuan Fu, Guan-Ting Lin, Hsiao-Tsung Hung, Hung-yi Lee

Organizations: National Taiwan University · ASUS Open Cloud Infrastructure Software Center · NVIDIA AI Technology Center

Abstract

End-to-end full-duplex Speech Language Models (SLMs) require precise turn-taking for natural interaction. However, optimizing temporal dynamics via standard raw-token reinforcement learning (RL) degrades semantic quality, causing severe generative collapse and repetition. We propose ASPIRin, an interactivity-optimized RL framework that explicitly decouples when to speak from what to say. Using Action Space Projection, ASPIRin maps the text vocabulary into a coarse-grained binary state (active speech vs. inactive silence). By applying Group Relative Policy Optimization (GRPO) with rule-based rewards, it balances user interruption and response latency. Empirical evaluations show ASPIRin optimizes interactivity across turn-taking, backchanneling, and pause handling. Crucially, isolating timing from token selection preserves semantic coherence and reduces the portion of duplicate n-grams by over 50% compared to standard GRPO, effectively eliminating degenerative repetition.

Figures & tables

Figure 1: Overview of the ASPIRin framework. (a) Action Space Projection & State Policy Optimization: The fine-grained text vocabulary is decoupled into a coarse-grained binary state (Active Speech vs. Inactive Silence) by grouping and summing non-padding and padding logits. This projected state policy is then explicitly optimized. (b) Rule-Based Rewards: The state policy is guided by continuous temporal constraints that penalize user interruption and excessive response latency. This explicit decoupling allows ASPIRin to master conversational timing without compromising semantic generation.

Dimension	Pause Handling		Backchannel			Smooth Turn Taking		User Interruption
Data	Synthetic	Candor	ICC			Candor		Synthetic
Metric	TOR ( $\downarrow$ )	TOR ( $\downarrow$ )	TOR ( $\downarrow$ )	Freq ( $\uparrow$ )	JSD ( $\downarrow$ )	TOR ( $\uparrow$ )	Latency ( $\downarrow$ )	TOR ( $\uparrow$ )	GPT-4o( $\uparrow$ )	Latency ( $\downarrow$ )
Moshi (w/o 3s prompt delay)	0.985	0.980	1.000	0.001	0.957	0.941	0.265	1.000	0.765	0.257
Moshi	0.467	0.495	0.436	0.044	0.705	0.748	0.161	0.901	3.894	1.159
Standard SFT	0.540	0.6389	0.927	0.0212	0.870	0.723	0.355	0.625	0.440	1.970
Standard GRPO	0.642	0.704	0.709	0.030	0.854	0.857	0.153	0.953	3.247	0.614

Table 1: Performance comparison of full-duplex models. We evaluate our proposed ASPIRin against Moshi baselines, Standard SFT, and Standard GRPO across four conversational dimensions: Pause Handling, Backchanneling, Smooth Turn-Taking, and User Interruption. Arrows (

\downarrow

\uparrow

) indicate whether lower or higher values indicate better performance. Latency is measured in seconds.

Panel of Figure 2 — Figure 2: Comparison of training reward dynamics between standard GRPO and ASPIRin.

	Example
Input Question	By the way, how about we discuss exercise routines instead?
Moshi	Sure, I’m good with that. What’s your routine look like? I’ve been trying to incorporate more cardio and strength framing into my routine. I’ve been doing a mix of running. (Rating: 4)
Standard SFT	Joyful Veal Eh is beneficial But I work in diverse inventions, so this actual definite divention of grapefish individual … (Rating: 0)
Standard GRPO	Hey, so I was just thinking, I have this friend, you know, who has a dog and you know, he says that his dog is uh you know really good you know at tracking and you know I was just wondering you know what you know what kind of you know … (Rating: 0)
ASPIRin (Ours)	Sure, that sounds good. Let’s start with the basic question. What are some of the most common types of exercises? Okay, so there are a lot of different types of exercises, but some of the most common ones are … (Rating: 5)

Table 2: Qualitative examples from the ”User Interruption” task. While the Standard SFT baseline hallucinates irrelevant vocabulary and Standard GRPO suffers from severe repetitive loops, ASPIRin successfully maintains semantic coherence and contextually appropriate responses, achieving parity with the base Moshi model.

Metric	seq-rep-n			Self-BLEU ( $\downarrow$ )
Metric	1-gram ( $\downarrow$ )	2-gram ( $\downarrow$ )	3-gram ( $\downarrow$ )	Self-BLEU ( $\downarrow$ )
Standard GRPO	0.303	0.117	0.072	0.369
ASPIRin (Ours)	0.202	0.054	0.029	0.343

Table 3: Evaluation of degenerative repetition using seq-rep-n and Self-BLEU. By isolating timing optimization from semantic token selection, ASPIRin effectively mitigates the degenerative repetition loops observed in standard GRPO, reducing 2-gram and 3-gram repetition by over 50%.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models
Jun 9, 2026Atsumoto Ohashi, Neil Zeghidour, Alexandre Défossez +1Spoken Dialogue Systems Multi-Objective Language Model Alignment
Kyutai, Paris, France · Gradium, Paris, France
HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Oct 6, 2026Kyudan Jung, Hyunsin Park, Yoonhyung Lee +5RL for Language Models Speech Language Models
Qualcomm AI Research · KAIST AI
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Jun 9, 2026J/K move · Enter open · S save

Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models

Atsumoto Ohashi, Neil Zeghidour, Alexandre Défossez, Eugene Kharitonov

Kyutai, Paris, France · Gradium, Paris, France

Spoken Dialogue Systems Multi-Objective Language Model Alignment Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Full-duplex spoken dialogue models can listen and speak simultaneously, making them a promising architecture for natural conversation. However, current models are trained solely with supervised learning through token-level likelihood maximization, which does not directly optimize interaction-level behaviors, causing interactivity issues such as excessive silence and ill-timed turn-taking. Recent work has applied reinforcement learning (RL) to improve interactivity, but existing methods address only a limited set of interactive behaviors in their rewards. In this work, we propose a post-training alignment method that comprehensively improves the interactivity of full-duplex spoken dialogue models through RL. We address the four canonical axes of interactivity: pause handling, turn-taking, backchanneling, and user interruption. For each axis, we extract short audio segments from human conversation corpora and optimize the model with axis-specific reward functions. An extra LLM-based reward for response quality prevents semantic degradation. We apply our method to two open-source models, Moshi and PersonaPlex, demonstrating consistent improvements in interactivity on both offline evaluation with pre-recorded audio and real-time multi-turn dialogue evaluation.

Abstract

Speech Language Models

Explore similar work

Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models

HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM