Abstract
Simultaneous speech translation must emit useful target text before the source is complete while preserving every committed token. We adapt a full-utterance speech language model using prefix supervision derived from its own complete- and partial-waveform translations, requiring neither transcripts nor human translations. We compare single-turn forced-prefix and multi-turn append-only decoding, use a confidence threshold to control the inference-time quality--latency trade-off, and vary the density of training prefixes with a separate synthesis margin. On FLEURS and CoVoST2 in three language directions, prefix training improves quality--latency frontiers over the unadapted model, and confidence provides the broadest consistently competitive operating range. Multi-turn decoding is generally stronger at low latency; under multi-turn training, commit-calibration error falls by 63--68% overall and 68--80% at early prefixes, whereas single-turn training provides only modest overall calibration gains and no early-prefix improvement. A small synthesis margin sometimes extends the frontier to lower latency, particularly on shorter utterances, while a larger margin degrades translation quality and calibration. Prefix adaptation therefore improves simultaneous speech translation, especially under multi-turn append-only decoding, while synthesis density introduces a non-monotonic quality--latency trade-off.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Explore similar work
Sep 5, 2026
In streaming simultaneous speech translation, the speech translation system is trained to learn a read-write policy that alternates between consuming source words and generating target ones. In a cascaded setting, the output from the speech recognizer is passed to a separate machine translation component, making it more difficult to learn such a policy. Approximations such as fixed wait-k strategies or target-suffix deletion can be employed, but these approaches do not provide the model with a streaming system's flexibility to make contextual read-write decisions. This paper presents a training strategy for a cascaded machine translation system that enables it to dynamically decide how much of the growing source prefix to translate. We achieve this by fine-tuning a large language model (Qwen3-8B) on stable prefixes of the training data, which are produced by pairing every source sentence prefix in the training data with the longest translation of that prefix that is shared with the full source sentence translation. We fine-tune variants of the model on different subsets of the prefixes and compare against wait-k and target-suffix deletion. We also investigate the effect of fine-tuning the target-token generation confidence. Our experiments show that stable prefixes improve the quality-latency tradeoff when translating from English into German, Japanese, and Chinese across a range of test sets.
Hieu Hoang, Amittai Axelrod, Matt Post
Microsoft
Apr 16, 2025
Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have shown strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on both in-domain (MuST-C) and out-of-domain (Europarl-ST) En-De and En-Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.
Biao Fu, Donglei Yu, Minpeng Liao +5
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
Sep 24, 2026
Large Language Models (LLMs) have shown strong performance in low-resource offline translation; however, extending them to simultaneous speech-to-speech translation (Simul-S2ST) remains challenging due to the scarcity of causally aligned training data with high cross-lingual speaker fidelity. In addition, existing approaches rely on fixed translation policy or confidence heuristics, leading to suboptimal quality and higher latency. We propose a causality-aware Simul-S2ST framework with a novel data pipeline that generates high-fidelity, causally aligned segments with improved voice transfer. The framework introduces (i) a factorized S2ST architecture (FAST), (ii) a causality-aware adaptive policy (CAP), and (iii) causality-aware latency metric. Experiments on CVSS Spanish, German, and French show that FAST-CAP consistently improves the quality-latency trade-off, achieving up to +1.2 BLEU and a 26% relative latency reduction over a fixed policy. Despite using substantially less training data than existing systems, FAST-CAP achieves state-of-the-art results in speech translation quality and speaker fidelity while yielding up to a 38.8% relative reduction in latency.
Amir Hussein, Enas Albasiri, Travis M. Bartley +7
Johns Hopkins University · NVIDIA