Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.
Figures & tables
Figure 1: Training and inference diagram of CRESiST. In the interleaved sequence, red and green target tokens mark whether the Router score at that position passes the decision threshold, i.e., whether the LLM body loss Lbody is computed there (tokens judged to be emitted too early are masked out). The Refiner loss Lrefiner is computed at every text position, and the two losses are summed; the same ground-truth token is thus the target of two prediction heads over one sequence, not duplicated in it.
Figure 2: Structural details of the RNN-T aligner and the Refiner layer.
Figure 3: Sentence-level quality-latency trade-off curves on the CoVoST2 and FLEURS datasets. The proposed framework consistently establishes a superior Pareto frontier compared to Seamless Streaming and AlignATT across all sub-tasks.
MuST-C
ACL 60/60
en-zh
en-de
en-zh
en-de
Method
COMET
LongYAAL
COMET
LongYAAL
COMET
LongYAAL
COMET
LongYAAL
Offline w/ Oracle Segmentation
Qwen2.5-Omni
79.41
/
91.47
/
84.42
/
91.72
/
CRESiST
79.11
/
91.35
/
83.36
/
90.03
/
Streaming w/ Oracle Segmentation
Table 1: Document-level simultaneous translation performance on the MuST-C and ACL 60/60 benchmarks. The COMET is evaluated through XCOMET-XXL. Our proposed CRESiST model achieves state-of-the-art translation quality with minimal continuous delay, demonstrating exceptional robustness in the challenging unsegmented streaming scenario. Oracle Segmentation denotes sentence-level segmentation taken from the native sentence timestamps of the test sets, i.e., inference is run independently on each human-segmented sentence.
Figure 4: Ablation on training strategies. “Direct SFT” uses RNN-T aligned data directly (similar to InfiniSST). “Stage 1 & 2” represents pipeline training without the final joint optimization. “CRESiST” indicates our full joint training framework, which yields the most significant latency reduction, particularly on resource-rich directions (e.g., zh ↔ en).
Figure 6
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Comparison of emission probabilities and decoding paths. The proposed RNN-T with axial attention (a) successfully maintains an incremental, monotonic alignment for simultaneous translation. In contrast, the conventional RNN-T (b) fails to emit tokens timely during the speech stream, causing its decoding path to delay and degenerate into an offline translation behavior at the end of the utterance.
ACL 60/60
MuST-C
System
COMET
LongYAAL
COMET
LongYAAL
SFT (InfiniSST-style)
68.43
2.27
66.16
2.34
SFT + GRPO (HPO recipe)
73.64
2.15
69.71
2.15
CRESiST
79.45
1.49
73.73
1.14
CRESiST + GRPO
80.04
1.37
78.47
1.23
Appendix
Table 3: Document-level En-Zh results (streaming w/o segmentation, XCOMET-XXL ↑ / LongYAAL ↓ ) with and without GRPO post-training. SFT denotes an InfiniSST-style supervised baseline on the same backbone.
En-Zh
Zh-En
Inference model
COMET
LAAL
COMET
LAAL
RNN-T Actor, before distillation
43.7
3.20
55.5
2.29
RNN-T Actor, after distillation
44.1
3.02
55.1
1.52
CRESiST (Speech LLM)
80.5
1.78
88.6
1.54
Appendix
Table 4: Using the RNN-T Actor directly as the inference model on FLEURS (XCOMET-XXL ↑ / LAAL ↓ ), before and after reverse distillation, compared with the full CRESiST Speech LLM.
Figure 7: Translation timing visualization for a sample from the FLEURS EN-ZH test set. The timeline illustrates the latency advantage of our model, which performs fine-grained, word-level adaptive translation, in contrast to the coarse-grained, clause-level buffering strategy employed by the Seed LiveInterpret 2.0 API.
Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have shown strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on both in-domain (MuST-C) and out-of-domain (Europarl-ST) En-De and En-Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.
Biao Fu, Donglei Yu, Minpeng Liao +5
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
Simultaneous speech translation (SimulST) requires incremental translation under strict latency constraints, yet remains challenging for decoder-only LLM systems due to limited context and cross-lingual reordering. Recent approaches often introduce architectural changes or explicit read/write policies to control output timing, which can be brittle in conversational speech where segmentation boundaries are ambiguous. We present a simple data-driven alternative: fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, and teacher-labeled prefix-to-prefix (P2P) targets with bounded waiting for fine-tuning, yielding CSSEL-P2P, where CSSEL is our proposed chunked streaming speech encoder LLM. In our in-house conversational speech evaluation, CSSEL-P2P improves streaming quality by +1.54 COMETKiwi over the CSSEL streaming baseline at comparable latency (+0.15s Average Lagging), suggesting effective SimulST without architectural changes via P2P supervision.
Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST and formulated it as a multi-turn dialogue task for efficient generation, achieving state-of-the-art performance. However, existing dialogue-style SimulST approaches typically rely on specialized training schemes to acquire simultaneous translation behavior, either using synthesized read/write trajectories, which may provide suboptimal training signals, or jointly optimizing translation quality and latency, whose competing objectives may cause optimization instability. In this paper, we introduce SimPOTrans, a reinforcement learning (RL) framework that enables offline speech translation models to self-adapt to dialogue-style SimulST. Through a simple two-stage RL post-training pipeline, SimPOTrans rewards an offline model for exhibiting translation behaviors beneficial to simultaneous scenarios when given partial speech inputs, thereby enabling offline-to-simultaneous self-adaptation without synthesized read/write trajectories or explicit latency optimization. At inference time, SimPOTrans requires no external decision policy and enables flexible quality--latency control by simply adjusting the chunk size of speech input. Experiments on the ACL 60/60 and MCIF benchmarks across three language directions (En→De, En→Zh, and En→Ja) demonstrate that SimPOTrans outperforms several strong baselines, achieving a superior quality--latency trade-off.