Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.
Figures & tables
Figure 1: Application overview of S3-S2ST. The system listens to partial source speech, decides when to commit, and produces translated speech while the source continues. The timeline is illustrative, not a measured latency trace.
Figure 2: Architecture of S3-S2ST. Chunkwise acoustic features and committed text form the Thinker context. Its final state supports text generation and an MLP READ/WRITE policy; a separate weighted fusion of eight layers conditions the Talker through a projection. Codec tokens are rendered by frozen flow matching and vocoder modules. READ advances the source; WRITE enables generation of a translation span. Token counts are schematic. Trainability of the other modules depends on the training stage.
System
Params.
BLEU ↑
COMET ↑
WER ↓
ASR-BLEU ↑
ASR-COMET ↑
SIM ↑
UTMOS ↑
FLEURS
SeamlessM4T-v2-large
2.3B †
26.18
0.800
7.34
24.80
0.729
0.026
3.16
Qwen3-Omni
30B/3B
34.46
0.905
2.54
31.71
0.843
0.015
4.16
UniSS (Quality)
1.5B
28.79
0.810
2.04
25.30
0.772
0.437
3.89
UniSS (Performance)
1.5B
27.64
0.790
2.36
24.13
0.756
0.438
3.90
S3-S2ST
4B
34.52
0.904
2.00
29.88
0.849
0.604
4.01
Table 1: Offline S2ST results on FLEURS and 9-pair. Bold and underline mark the best and second-best scores.
System
Params.
BLEU ↑
COMET ↑
WER ↓
ASR-BLEU ↑
ASR-COMET ↑
SIM ↑
UTMOS ↑
FLEURS
IWSLT (VAD)
2.3B †
21.16
0.665
–
–
–
–
–
SeamlessStreaming
2.5B †
23.17
0.730
13.17
21.80
0.638
0.034
3.36
X-Translator
8B
28.63
0.759
5.40
24.87
0.699
0.547
3.19
SimulS2ST-Omni (m2)
3B
26.46
0.743
11.87
21.27
0.626
0.561
2.96
Hy-SimulMT (1.6 s)
30B/3B
30.20
0.845
2.28
26.24
0.791
0.396
3.72
Table 2: Streaming translation results on FLEURS and 9-pair. “–” denotes inapplicable speech metrics.
Figure 3: Quality–latency tradeoff on RealSI, averaged equally across directions. Labels denote our fixed thresholds, SimulS2ST-Omni latency multipliers m1–m5, and Hy-SimulMT subsequent chunk durations. The star denotes our playback-adaptive policy.
Frontend recipe
COMET ↑
Earlier block-causal recipe
0.687
640 ms chunkwise recipe
0.827
Table 3: S2TT-stage frontend recipes on FLEURS ( τ=0.5 ). Training recipes differ.
Policy
COMET ↑
ASR-COMET ↑
StartOffset ↓
FLEURS
Fixed 6
0.782
0.729
5.55
Fixed 8
0.814
0.752
6.83
MLP
0.828
0.767
6.61
Table 4: Commitment policies on FLEURS. Fixed K writes every K source blocks; MLP uses τ=0.5 . StartOffset is in seconds.
Policy
ASR-BLEU ↑
ASR-COMET ↑
StartOffset ↓
Fixed 0.1
25.02
0.737
4.90
Dynamic
25.74
0.751
4.97
Table 5: Fixed versus playback-adaptive thresholds on FLEURS. StartOffset is in seconds.
Conditioning
ASR-COMET ↑
WER ↓
Final layer
0.744
4.13
Eight layers
0.751
3.74
Table 6: Speech-conditioning ablation on FLEURS with the dynamic policy.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Procedure: Streaming data construction
Input: Speech pair (x,x) , transcripts (u,v) , direction d , and training stage.
Output: A validated READ/WRITE trajectory, or rejection.
1
Index lossless text units in u and v .
2
D←QwenDraft(u,v,d) : propose paired monotonic endpoints with source-prefix support.
3
D←QwenRevise(u,v,D,d) : revise into exact original-text substrings.
4
Merge punctuation-only mismatches; check paired nonempty spans, transcript reconstruction, and review flags. Retry or reject failures.
Appendix
Table 10
Substage
Initialization
Updated modules
Training data
Updates
Batch size
Learning rate
I. Streaming S2TT adaptation
Encoder adaptation
Pretrained S2TT components
Encoder, adaptor, Thinker LoRA
∼ 3M h
10k
128
LoRA: 10−5 ; encoder/adaptor: 2×10−5
Commitment learning
Encoder-adapted model with LoRA merged
Thinker, READ/WRITE gate
∼ 17k h
10k
8
Thinker: 5×10−6 ; gate: 10−4
II. Offline S2ST adaptation
Talker adaptation
Pretrained source frontend, Thinker, and Talker; new speech interface
Talker, speech interface
∼ 300k h
15k
64
Talker: 5×10−6 ; projection/fusion: 10−4
III. Streaming S2ST adaptation
Appendix
Table 7: Adaptation stages used to obtain the reported models.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only ∼2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.
Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate - for example, whether they process continuous speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental). For instance, SimulEval, the most widely used framework, supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. As a result, comparing systems fairly and consistently across studies remains challenging, with no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.