Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation

Hieu Hoang; Amittai Axelrod

cs.CLOct 2, 2026

Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation

Authors: Hieu Hoang, Amittai Axelrod

Organizations: Microsoft

Abstract

Simultaneous speech translation must emit useful target text before the source is complete while preserving every committed token. We adapt a full-utterance speech language model using prefix supervision derived from its own complete- and partial-waveform translations, requiring neither transcripts nor human translations. We compare single-turn forced-prefix and multi-turn append-only decoding, use a confidence threshold to control the inference-time quality--latency trade-off, and vary the density of training prefixes with a separate synthesis margin. On FLEURS and CoVoST2 in three language directions, prefix training improves quality--latency frontiers over the unadapted model, and confidence provides the broadest consistently competitive operating range. Multi-turn decoding is generally stronger at low latency; under multi-turn training, commit-calibration error falls by 63--68% overall and 68--80% at early prefixes, whereas single-turn training provides only modest overall calibration gains and no early-prefix improvement. A small synthesis margin sometimes extends the frontier to lower latency, particularly on shorter utterances, while a larger margin degrades translation quality and calibration. Prefix adaptation therefore improves simultaneous speech translation, especially under multi-turn append-only decoding, while synthesis density introduces a non-monotonic quality--latency trade-off.

Figures & tables

Language	Training dataset	$\delta=0$	$\delta=0.2$	$\delta=0.5$
EN $\rightarrow$ DE	LibriSpeech-100	10.2	15.9	18.9
	LibriSpeech-360	10.2	16.0	18.9
	Common Voice	4.2	5.5	6.5
EN $\rightarrow$ JA	LibriSpeech-100	16.7	35.4	46.4
	LibriSpeech-360	16.6	35.3	46.1
	Common Voice	9.5	13.9	20.5

Table 1: Effect of the synthesis margin on the training data, measured by the mean cumulative target-prefix length at each non-final

2

s audio block.

Panel of Figure 1 — Figure 1: Quality–latency frontiers for the base and canonical single-turn models under confidence, wait- $k$ , and suffix-deletion controls.

Direction	Model	wait- $k$	suf. del.	confidence
FLEURS
EN $\rightarrow$ DE	base	54.0	54.8	52.2
	trained	56.0	57.0	57.0
EN $\rightarrow$ JA	base	75.6	71.6	73.3
	trained	76.6	76.3	78.4
EN $\rightarrow$ ZH	base	41.7	41.7	40.5

Table 2: AUC for the base and single-turn models in Figure 1 over each panel’s shared latency interval.

Panel of Figure 2 — Figure 2: Quality–latency frontiers for the base and canonical multi-turn models.

Direction	Model	wait- $k$	suf. del.	confidence
FLEURS
EN $\rightarrow$ DE	base	49.7	48.7	48.6
	trained	57.2	59.0	66.4
EN $\rightarrow$ JA	base	64.5	61.1	61.0
	trained	68.8	71.5	78.4
EN $\rightarrow$ ZH	base	39.2	39.5	39.4

Table 3: AUC for the base and multi-turn models in Figure 2 over each panel’s shared latency interval.

Panel of Figure 3 — Figure 3: Confidence-controlled multi-turn frontiers for models trained with synthesis margins.

Panel of Figure 4 — Figure 4: Confidence frontiers for the canonical single-turn and multi-turn models and the multi-turn $\delta=0.2$ synthesis-margin ablation.

FLEURS
	single-turn	multi-turn	$\delta=0.2$
EN $\rightarrow$ DE	58.9	65.9	67.1
EN $\rightarrow$ JA	80.0	80.1	79.2
EN $\rightarrow$ ZH	44.8	45.2	45.3
CoVoST 2
EN $\rightarrow$ DE	67.4	72.9	74.5

Table 4: AUC corresponding to Figure 4 . Each row uses the shared latency interval for that dataset–language panel; best per row in bold .

Figure 5: Commit-confidence ECE for the single-turn, multi-turn, and synthesis-margin models. Error bars are pointwise $95\%$ utterance-cluster bootstrap intervals.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset / genre	Reference	$N$	Train mix	Avg. audio (s)
Training
LibriSpeech-100 Audiobook speech	Self-teacher	$27{,}403$	$20.0\%$	$12.7$
LibriSpeech-360 Audiobook speech	Self-teacher	$68{,}469$	$50.0\%$	$12.6$
Common Voice Crowd-sourced read speech	Self-teacher	$41{,}018$	$30.0\%$	$5.7$
Training total		$136{,}890$	$100.0\%$	—
Validation

Appendix

Table 5: Training, validation, and test data.

Direction	single-turn	multi-turn
EN $\rightarrow$ DE	2,000 / 6,000 (384k)	3,000 / 6,000 (384k)
EN $\rightarrow$ JA	2,500 / 6,000 (384k)	5,500 / 6,000 (384k)
EN $\rightarrow$ ZH	1,000 / 6,000 (384k)	3,000 / 3,000 (192k)

Appendix

Table 6: Selected update / final update evaluated; parentheses give the total number of optimized records through the final update (updates multiplied by effective batch size 64).

Explore similar work

Dynamic Lagging using Stable-Prefix Training for Simultaneous Translation
Sep 5, 2026Hieu Hoang, Amittai Axelrod, Matt PostMachine Translation Simultaneous Speech Translation
Microsoft
Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
Apr 16, 2025Biao Fu, Donglei Yu, Minpeng Liao +5Speech Translation Simultaneous Speech Translation
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation
Sep 24, 2026Amir Hussein, Enas Albasiri, Travis M. Bartley +7Speech Translation Simultaneous Speech Translation
Johns Hopkins University · NVIDIA

Sep 5, 2026J/K move · Enter open · S save

Dynamic Lagging using Stable-Prefix Training for Simultaneous Translation

Hieu Hoang, Amittai Axelrod, Matt Post

Microsoft

Machine Translation Simultaneous Speech Translation

Open paper AskPDF arXiv

In streaming simultaneous speech translation, the speech translation system is trained to learn a read-write policy that alternates between consuming source words and generating target ones. In a cascaded setting, the output from the speech recognizer is passed to a separate machine translation component, making it more difficult to learn such a policy. Approximations such as fixed wait-k strategies or target-suffix deletion can be employed, but these approaches do not provide the model with a streaming system's flexibility to make contextual read-write decisions. This paper presents a training strategy for a cascaded machine translation system that enables it to dynamically decide how much of the growing source prefix to translate. We achieve this by fine-tuning a large language model (Qwen3-8B) on stable prefixes of the training data, which are produced by pairing every source sentence prefix in the training data with the longest translation of that prefix that is shared with the full source sentence translation. We fine-tune variants of the model on different subsets of the prefixes and compare against wait-k and target-suffix deletion. We also investigate the effect of fine-tuning the target-token generation confidence. Our experiments show that stable prefixes improve the quality-latency tradeoff when translating from English into German, Japanese, and Chinese across a range of test sets.

Abstract

Explore similar work

Dynamic Lagging using Stable-Prefix Training for Simultaneous Translation

Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation