CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation
Organizations: Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · University of Macau · University College London
Abstract
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.
Figures & tables
| MuST-C | ACL 60/60 | |||||||
| en-zh | en-de | en-zh | en-de | |||||
| Method | COMET | LongYAAL | COMET | LongYAAL | COMET | LongYAAL | COMET | LongYAAL |
| Offline w/ Oracle Segmentation | ||||||||
| Qwen2.5-Omni | 79.41 | / | 91.47 | / | 84.42 | / | 91.72 | / |
| CRESiST | 79.11 | / | 91.35 | / | 83.36 | / | 90.03 | / |
| Streaming w/ Oracle Segmentation | ||||||||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| ACL 60/60 | MuST-C | |||
|---|---|---|---|---|
| System | COMET | LongYAAL | COMET | LongYAAL |
| SFT (InfiniSST-style) | 68.43 | 2.27 | 66.16 | 2.34 |
| SFT + GRPO (HPO recipe) | 73.64 | 2.15 | 69.71 | 2.15 |
| CRESiST | 79.45 | 1.49 | 73.73 | 1.14 |
| CRESiST + GRPO | 80.04 | 1.37 | 78.47 | 1.23 |
| En-Zh | Zh-En | |||
|---|---|---|---|---|
| Inference model | COMET | LAAL | COMET | LAAL |
| RNN-T Actor, before distillation | 43.7 | 3.20 | 55.5 | 2.29 |
| RNN-T Actor, after distillation | 44.1 | 3.02 | 55.1 | 1.52 |
| CRESiST (Speech LLM) | 80.5 | 1.78 | 88.6 | 1.54 |
| Asset Name | Type | License | Reference | URL |
| Datasets | ||||
| Emilia | Dataset | CC BY 4.0 | [ 29 ] | https://huggingface.co/datasets/amphion/Emilia-Dataset |
| WenetSpeech | Dataset | CC BY 4.0 | [ 30 ] | https://www.openslr.org/121/ |
| GigaSpeech | Dataset | Apache 2.0 | [ 31 ] | https://github.com/SpeechColab/GigaSpeech |
| FLEURS | Dataset | CC BY 4.0 | [ 33 ] | https://huggingface.co/datasets/google/fleurs |
| CoVoST 2 | Dataset | CC BY-NC 4.0 | [ 34 ] | https://huggingface.co/datasets/facebook/covost2 |
Simultaneous Speech Translation
Simultaneous speech translation converts speech into another language as audio arrives, balancing translation quality, latency, and fluent delivery.
See all 30 papers