Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST and formulated it as a multi-turn dialogue task for efficient generation, achieving state-of-the-art performance. However, existing dialogue-style SimulST approaches typically rely on specialized training schemes to acquire simultaneous translation behavior, either using synthesized read/write trajectories, which may provide suboptimal training signals, or jointly optimizing translation quality and latency, whose competing objectives may cause optimization instability. In this paper, we introduce SimPOTrans, a reinforcement learning (RL) framework that enables offline speech translation models to self-adapt to dialogue-style SimulST. Through a simple two-stage RL post-training pipeline, SimPOTrans rewards an offline model for exhibiting translation behaviors beneficial to simultaneous scenarios when given partial speech inputs, thereby enabling offline-to-simultaneous self-adaptation without synthesized read/write trajectories or explicit latency optimization. At inference time, SimPOTrans requires no external decision policy and enables flexible quality--latency control by simply adjusting the chunk size of speech input. Experiments on the ACL 60/60 and MCIF benchmarks across three language directions (En→De, En→Zh, and En→Ja) demonstrate that SimPOTrans outperforms several strong baselines, achieving a superior quality--latency trade-off.
Figures & tables
Figure 1: Overview of the SimPOTrans training pipeline. (1) Stage 0: Offline Translation Pretraining (SFT). SFT establishes the offline translation capability of the LLM-based speech translation model. (2) Stage 1: Offline Translation Optimization (RL). RL-Stage I uses GRPO with a translation-quality reward to further enhance offline translation quality on complete utterances. (3) Stage 2: Offline-to-Simultaneous Translation Adaptation (RL). RL-Stage II exposes the offline-trained model to partial speech inputs in a multi-turn setting and optimizes it with quality and monotonicity rewards together with a repetition penalty, enabling the model to self-adapt to dialogue-style SimulST.
Figure 2: Comparison of quality–latency trade-offs on the ACL 60/60 benchmark for En → De, En → Zh, and En → Ja. Rows report COMET, MetricX-24, and BLEU, respectively, with latency measured by LongYAAL. MetricX-24 scores are negated so that higher values consistently indicate better translation quality across all metrics. The SimPOTrans curves are obtained using chunk sizes of {1.0,1.5,2.0,3.0,4.0,5.0} seconds.
Lang.
System
COMET
LongYAAL
PDF
En → De
NeMo *
0.93
2.0
√
CUHKSZ
0.85
1.5
√
PINCH-AST *
0.84
1.8
×
CUNI-POCKET
0.77
1.7
×
Ours
0.87
1.6
×
En → Zh
CUHKSZ
0.74
1.5
√
Table 1: Comparison with IWSLT 2026 Simultaneous Track systems on the MCIF development set in the low-latency regime (≤2s) . LongYAAL is reported in seconds. Asterisks indicate cascaded systems. PDF indicates whether PDF-derived extra context is used.
Figure 3: Ablation studies. (a) Comparison of quality–latency trade-offs across different training stages. (b) Effects of the monotonicity reward (Mono.) and repetition penalty (Rep.) in RL-Stage II. (c) Effect of the speech chunk size in RL-Stage II.
Figure 4: Monotonicity across SimPOTrans training stages under simultaneous inference with a fixed chunk size of 1.5 seconds.
Figure 5: An En → Zh example comparing translation behaviors before and after offline-to-simultaneous self-adaptation. Colored spans and connecting lines indicate source–target alignments. The offline model largely follows the reference word order, which involves substantial reordering relative to the source. In contrast, SimPOTrans conveys the core meaning in an order closer to the source, producing a more monotonic translation better suited to simultaneous generation.
Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have shown strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on both in-domain (MuST-C) and out-of-domain (Europarl-ST) En-De and En-Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.
Biao Fu, Donglei Yu, Minpeng Liao +5
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems formulate SimulST as a multi-turn dialogue task, requiring specialized interleaved training data and relying on computationally expensive large language model (LLM) inference for decision-making. In this paper, we propose SimulSense, a novel framework for SimulST that mimics human interpreters by continuously reading input speech and triggering write decisions to produce translation when a new sense unit is perceived. Experiments against two state-of-the-art baseline systems demonstrate that our proposed method achieves a superior quality-latency tradeoff and substantially improved real-time efficiency, where its decision-making is up to 9.6x faster than the baselines.
Simultaneous speech-to-text translation (SimulST) generates translations while speech is still unfolding, requiring a streaming policy that decides when to read and when to write. State-of-the-art approaches rely on attention-based encoder-decoder models where cross-attention provides explicit alignment signals. In contrast, Speech Large Language Models (SpeechLLMs) are decoder-only architectures relying solely on self-attention. This raises a central question: whether decoder self-attention contains sufficiently stable alignment signals to guide the streaming policy. Moreover, existing approaches typically rely on training-based adaptations or heuristic wait-k policies and have not been validated in long-form settings. To fill these gaps, we propose Decoder-Only Attention (DOA), a training-free policy that enables long-form simultaneous translation with off-the-shelf SpeechLLMs by deriving a proxy alignment from self-attention. Experiments on Phi4-Multimodal and Qwen3-Omni show that DOA provides an effective alignment signal for supporting streaming decisions, enabling low-latency long-form SimulST with quality close to offline decoding without retraining.