DIAL-S2ST: Learning Latency Control for Simultaneous Speech-to-Speech Translation

Jeongsoo Choi; Junseok Ahn; Joon Son Chung

cs.CLOct 23, 2026

DIAL-S2ST: Learning Latency Control for Simultaneous Speech-to-Speech Translation

Authors: Jeongsoo Choi, Junseok Ahn, Joon Son Chung

Organizations: KAIST

Abstract

We propose DIAL-S2ST, a post-training method for learning inference-time latency control in simultaneous speech-to-speech translation models. Our approach reuses pretrained translation capabilities through self-distillation from lookahead predictions, while limiting the influence of large text distillation losses to account for the imitation gap. This provides supervision for earlier emission without paired target data or reward computation. To make the learned timing adjustable at inference, we jointly train a single LoRA adapter across scales with matched lookahead guidance. Experiments on French-to-English translation with Hibiki demonstrate that only 10 hours of source speech enable this adaptation, providing flexible latency control with a favorable quality-latency trade-off.

Figures & tables

Figure 1: Overview of DIAL-S2ST training and inference.

Figure 2: Quality-latency trade-off on the CVSS Fr

\to

En test set. Connected points represent different inference-time latency settings of the same model, while isolated points correspond to distinct models.

	LAAL (s)
	1.8	2.3	2.8
Ours	32.5	34.8	35.6
w/ self-training (shifted rollouts)	32.0	33.2	34.3
w/ off-policy (teacher rollouts)	31.1	34.6	35.5

Table 1: ASR-BLEU at matched LAAL on the CVSS Fr

\to

En test set.

	ASR-BLEU	LAAL	End	Speaker	UTMOS
	$\uparrow$	$\downarrow$	Offset $\downarrow$	Sim. $\uparrow$	$\uparrow$
Ours	34.4	2.16	1.73	0.391	3.41
w/o text KL masking	33.1	2.20	1.73	0.390	3.40
w/o acoustic KL	34.1	2.08	1.75	0.385	3.41
w/o speech KL	32.7	2.16	1.69	0.337	3.33
w/ grouped text KL	34.2	2.22	1.94	0.392	3.42

Table 2: Ablations of DIAL-S2ST (

\alpha=0.5

Speech Translation

Speech translation converts spoken language into text or speech in another language, including real-time systems that balance fidelity and latency.

See all 60 papers

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation
Sep 24, 2026Amir Hussein, Enas Albasiri, Travis M. Bartley +7Speech Translation Simultaneous Speech Translation
Johns Hopkins University · NVIDIA
SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning
Oct 26, 2026Haotian Tan, Hiroki Ouchi, Sakriani SaktiSimultaneous Speech Translation Speech Language Models
Nara Institute of Science and Technology, Japan
Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation
Oct 2, 2026Hieu Hoang, Amittai AxelrodSpeech Translation Simultaneous Speech Translation
Microsoft

Sep 24, 2026J/K move · Enter open · S save

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation

Amir Hussein, Enas Albasiri, Travis M. Bartley, Nourchene Ferchichi+6

Johns Hopkins University · NVIDIA

Speech Translation Simultaneous Speech Translation

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong performance in low-resource offline translation; however, extending them to simultaneous speech-to-speech translation (Simul-S2ST) remains challenging due to the scarcity of causally aligned training data with high cross-lingual speaker fidelity. In addition, existing approaches rely on fixed translation policy or confidence heuristics, leading to suboptimal quality and higher latency. We propose a causality-aware Simul-S2ST framework with a novel data pipeline that generates high-fidelity, causally aligned segments with improved voice transfer. The framework introduces (i) a factorized S2ST architecture (FAST), (ii) a causality-aware adaptive policy (CAP), and (iii) causality-aware latency metric. Experiments on CVSS Spanish, German, and French show that FAST-CAP consistently improves the quality-latency trade-off, achieving up to +1.2 BLEU and a 26% relative latency reduction over a fixed policy. Despite using substantially less training data than existing systems, FAST-CAP achieves state-of-the-art results in speech translation quality and speaker fidelity while yielding up to a 38.8% relative reduction in latency.

Abstract

Speech Translation

Explore similar work

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation

SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning

Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation