Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.
Figures & tables
Figure 1: SimulU Overview. The incoming speech input, here in English, is represented by the blue waveform , while the speech output, here in Italian, is represented by the black waveform .
Word History = 5
Word History = 10
Word History = 15
Word History = 20
Word History = 25
ASR
Start
End
ASR
Start
End
ASR
Start
End
ASR
Start
End
ASR
Start
End
BLEU
Offset
Offset
BLEU
Offset
Offset
BLEU
Offset
Offset
BLEU
Offset
Offset
BLEU
Offset
Offset
2
17.97
0.90
77.24
18.56
0.90
128.68
19.38
0.90
161.47
16.60
0.90
187.27
13.63
0.90
183.51
4
16.93
1.36
51.18
23.04
1.24
147.60
18.65
1.36
151.84
18.15
1.36
240.88
12.73
1.36
202.10
6
16.54
1.72
55.95
18.85
1.72
131.39
20.32
1.72
179.22
17.87
1.72
256.14
11.37
1.72
184.63
8
18.38
1.77
46.78
17.64
1.77
121.12
20.12
1.77
189.09
18.25
1.77
263.51
11.37
1.77
173.28
Table 1: ASR-BLEU scores for different Word History ( WH ) configurations (5-25) across combinations of cut-off frame f (2, 4, 6, 8).
Figure 2: ASR-BLEU and Start Offset scores across different systems for each language pair of MuST-C v1 tst-COMMON.
TTS sys.
Metric
de
fr
it
nl
pt
es
ru
ro
Seam.TTS
WER (%)
41.77
35.28
39.03
40.81
43.29
34.59
46.89
40.15
CER (%)
31.76
28.75
31.29
31.33
34.41
29.14
35.56
26.29
UTMOS
2.82
2.60
3.63
3.50
3.87
3.60
3.32
3.83
( ± std)
(0.32)
(0.35)
(0.34)
(0.27)
(0.33)
(0.33)
(0.41)
(0.29)
XTTS-v2
WER (%)
9.789
14.02
10.25
10.19
6.24
3.93
9.78
—
CER (%)
7.70
10.60
7.85
7.00
3.17
2.38
6.49
—
Table 2: TTS results for the Seam.TTS and XTTS-v2 systems.
System
de
fr
it
nl
pt
es
ru
ro
SimulU
247
224
100
82
146
106
106
34
(137)
(132)
(6)
(9)
(6)
(5)
(13)
(2)
Top
246
224
262
40
251
217
68
58
Cascade
(137)
(132)
(122)
(44)
(123)
(124)
(62)
(52)
Table 3: End Offset in milliseconds (mean and std) for SimulU and the top cascade for each language averaged across cutoff frame f and speech segment size, respectively.
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab