S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation

Yuxiang Zhao; Zhiyun Fan; Yu Pu; Yanjie An; Yushen Chen; Wenrui Liang; Baiji Liu; Yunchong Xiao

cs.SDOct 27, 2026

S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation

Authors: Yuxiang Zhao, Zhiyun Fan, Yu Pu, Yanjie An, Yushen Chen, Wenrui Liang, Baiji Liu, Yunchong Xiao,

Organizations: 1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · 2Shanghai Innovation Institute · 3Alibaba Token Foundry · 4Independent Researcher

Abstract

Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.

Figures & tables

Figure 1: Application overview of S3-S2ST. The system listens to partial source speech, decides when to commit, and produces translated speech while the source continues. The timeline is illustrative, not a measured latency trace.

Figure 2: Architecture of S3-S2ST. Chunkwise acoustic features and committed text form the Thinker context. Its final state supports text generation and an MLP READ/WRITE policy; a separate weighted fusion of eight layers conditions the Talker through a projection. Codec tokens are rendered by frozen flow matching and vocoder modules. READ advances the source; WRITE enables generation of a translation span. Token counts are schematic. Trainability of the other modules depends on the training stage.

FLEURS
System	Params.	BLEU $\uparrow$	COMET $\uparrow$	WER $\downarrow$	ASR-BLEU $\uparrow$	ASR-COMET $\uparrow$	SIM $\uparrow$	UTMOS $\uparrow$
SeamlessM4T-v2-large	2.3B †	26.18	0.800	7.34	24.80	0.729	0.026	3.16
Qwen3-Omni	30B/3B	34.46	0.905	2.54	31.71	0.843	0.015	4.16
UniSS (Quality)	1.5B	28.79	0.810	2.04	25.30	0.772	0.437	3.89
UniSS (Performance)	1.5B	27.64	0.790	2.36	24.13	0.756	0.438	3.90
S3-S2ST	4B	34.52	0.904	2.00	29.88	0.849	0.604	4.01

Table 1: Offline S2ST results on FLEURS and 9-pair. Bold and underline mark the best and second-best scores.

FLEURS
System	Params.	BLEU $\uparrow$	COMET $\uparrow$	WER $\downarrow$	ASR-BLEU $\uparrow$	ASR-COMET $\uparrow$	SIM $\uparrow$	UTMOS $\uparrow$
IWSLT (VAD)	2.3B †	21.16	0.665	–	–	–	–	–
SeamlessStreaming	2.5B †	23.17	0.730	13.17	21.80	0.638	0.034	3.36
X-Translator	8B	28.63	0.759	5.40	24.87	0.699	0.547	3.19
SimulS2ST-Omni (m2)	3B	26.46	0.743	11.87	21.27	0.626	0.561	2.96
Hy-SimulMT (1.6 s)	30B/3B	30.20	0.845	2.28	26.24	0.791	0.396	3.72

Table 2: Streaming translation results on FLEURS and 9-pair. “–” denotes inapplicable speech metrics.

Figure 3: Quality–latency tradeoff on RealSI, averaged equally across directions. Labels denote our fixed thresholds, SimulS2ST-Omni latency multipliers m1–m5, and Hy-SimulMT subsequent chunk durations. The star denotes our playback-adaptive policy.

Frontend recipe	COMET $\uparrow$
Earlier block-causal recipe	0.687
640 ms chunkwise recipe	0.827

Table 3: S2TT-stage frontend recipes on FLEURS (

\tau=0.5

). Training recipes differ.

Policy	COMET $\uparrow$	ASR-COMET $\uparrow$	StartOffset $\downarrow$
FLEURS
Fixed $6$	0.782	0.729	5.55
Fixed $8$	0.814	0.752	6.83
MLP	0.828	0.767	6.61

Table 4: Commitment policies on FLEURS. Fixed

K

writes every

K

source blocks; MLP uses

\tau=0.5

. StartOffset is in seconds.

Policy	ASR-BLEU $\uparrow$	ASR-COMET $\uparrow$	StartOffset $\downarrow$
Fixed $0.1$	25.02	0.737	4.90
Dynamic	25.74	0.751	4.97

Table 5: Fixed versus playback-adaptive thresholds on FLEURS. StartOffset is in seconds.

Conditioning	ASR-COMET $\uparrow$	WER $\downarrow$
Final layer	0.744	4.13
Eight layers	0.751	3.74

Table 6: Speech-conditioning ablation on FLEURS with the dynamic policy.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Procedure: Streaming data construction
	Input: Speech pair $(x,\widetilde{x})$ , transcripts $(u,v)$ , direction $d$ , and training stage.
	Output: A validated READ/WRITE trajectory, or rejection.
1	Index lossless text units in $u$ and $v$ .
2	$D\leftarrow\operatorname{QwenDraft}(u,v,d)$ : propose paired monotonic endpoints with source-prefix support.
3	$D\leftarrow\operatorname{QwenRevise}(u,v,D,d)$ : revise into exact original-text substrings.
4	Merge punctuation-only mismatches; check paired nonempty spans, transcript reconstruction, and review flags. Retry or reject failures.

Appendix

Table 10

Substage	Initialization	Updated modules	Training data	Updates	Batch size	Learning rate
I. Streaming S2TT adaptation
Encoder adaptation	Pretrained S2TT components	Encoder, adaptor, Thinker LoRA	$\sim$ 3M h	10k	128	LoRA: $10^{-5}$ ; encoder/adaptor: $2\times 10^{-5}$
Commitment learning	Encoder-adapted model with LoRA merged	Thinker, READ/WRITE gate	$\sim$ 17k h	10k	8	Thinker: $5\times 10^{-6}$ ; gate: $10^{-4}$
II. Offline S2ST adaptation
Talker adaptation	Pretrained source frontend, Thinker, and Talker; new speech interface	Talker, speech interface	$\sim$ 300k h	15k	64	Talker: $5\times 10^{-6}$ ; projection/fusion: $10^{-4}$
III. Streaming S2ST adaptation

Appendix

Table 7: Adaptation stages used to obtain the reported models.

Simultaneous Speech Translation

Simultaneous speech translation converts speech into another language as audio arrives, balancing translation quality, latency, and fluent delivery.

See all 26 papers

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
Jul 22, 2026Rongshen He, Xinyu Liang, Dekun Chen +3Speech Translation Simultaneous Speech Translation
1The Chinese University of Hong Kong, Shenzhen
Streaming Speech-to-Text Translation with a SpeechLLM
May 14, 2026Titouan Parcollet, Shucong Zhang, Xianrui Zheng +1Speech Translation Simultaneous Speech Translation
Samsung, AI Center – Cambridge, United Kingdom
Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems
Dec 19, 2025Marco Gaido, Sara Papi, Mauro Cettolo +2Speech Translation Simultaneous Speech Translation
Fondazione Bruno Kessler, Trento, Italy

Jul 22, 2026J/K move · Enter open · S save

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

Rongshen He, Xinyu Liang, Dekun Chen, Jiaqi Li+2

1The Chinese University of Hong Kong, Shenzhen

Speech Translation Simultaneous Speech Translation Speech Language Models

Open paper AskPDF arXiv

Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only

\sim

2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.

Abstract

Simultaneous Speech Translation

Explore similar work

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

Streaming Speech-to-Text Translation with a SpeechLLM

Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems