Large Language Models (LLMs) have shown strong performance in low-resource offline translation; however, extending them to simultaneous speech-to-speech translation (Simul-S2ST) remains challenging due to the scarcity of causally aligned training data with high cross-lingual speaker fidelity. In addition, existing approaches rely on fixed translation policy or confidence heuristics, leading to suboptimal quality and higher latency. We propose a causality-aware Simul-S2ST framework with a novel data pipeline that generates high-fidelity, causally aligned segments with improved voice transfer. The framework introduces (i) a factorized S2ST architecture (FAST), (ii) a causality-aware adaptive policy (CAP), and (iii) causality-aware latency metric. Experiments on CVSS Spanish, German, and French show that FAST-CAP consistently improves the quality-latency trade-off, achieving up to +1.2 BLEU and a 26% relative latency reduction over a fixed policy. Despite using substantially less training data than existing systems, FAST-CAP achieves state-of-the-art results in speech translation quality and speaker fidelity while yielding up to a 38.8% relative reduction in latency.
Figures & tables
Es-En
De-En
Fr-En
UTMOS
Spk-Sim
UTMOS
Spk-Sim
UTMOS
Spk-Sim
CVSS-T (public)
2.4
0.22
2.5
0.26
2.5
0.25
CVSS-T (ours)
2.4
0.64
2.5
0.61
2.5
0.57
TABLE I: Comparison of speaker similarity and audio quality on the CVSS-T Dataset, with audio quality measured by UTMOS-V2.
Fig. 1: Overview of the data generation pipeline. The process includes (1) alignments generation, (2) unique mapping and causal pivot extraction, and (3) causality-aware adaptive chunks construction.
Fig. 2: Overview of the proposed FAST architecture with multistream joint sequence modeling.
Fig. 3: Illustration of LAAL and CAAL computation. Oracle delays (green) and system delays (blue) are paired according to the alignment mapping (arrows), and lagging values (red) are computed as the difference between each system delay and its aligned oracle delay.
Text Translation
ASR Transcript
#
System
LLM
Align.
BLEU
chrF
COMET
BLEU
chrF
COMET
1
FAST (IL)
Qwen
NFA
18.7
49.0
68.6
12.5
39.5
61.1
2
FAST (IL)
Qwen
MFA
23.1
49.7
70.0
13.5
40.2
62.5
3
FAST-Lat (MS)
Qwen
MFA
26.6
54.8
75.4
4.5
23.3
56.0
4
FAST (MS)
Qwen
MFA
26.7
54.6
75.5
20.8
48.6
68.2
5
FAST (MS)
Llama
MFA
25.1
54.3
75.4
19.9
48.0
67.5
TABLE II: Ablation study on the CVSS-T development set comparing alignment methods (NFA vs. MFA), multimodal representations (interleaved, IL, vs. multistream, MS), and TTS conditioning (text vs. LLM latent representations, Lat). Translation quality is evaluated with BLEU, chrF++, and COMET, with speech metrics computed on ASR transcriptions.
Avg.
Latency (s)
Translation Quality
System
input (s)
CAAL ↓
LAAL ↓
BLEU ↑
chrF ↑
COMET ↑
FAST-CAP (Sp-En)
0.4
0.33
0.57
12.1
47.0
67.1
FAST-CAP (Sp-En)
0.9
0.52
0.88
24.0
55.3
75.0
FAST-CAP (Sp-En)
1.5
0.80
1.15
27.8
57.6
78.0
FAST-Fixed (Sp-En)
2.0
0.87
1.17
26.7
54.6
75.5
FAST-CAP (De-En)
1.5
0.68
1.20
34.6
62.0
78.1
TABLE III: Comparison of FAST on CVSS-T development set using the causality-aware adaptive policy (CAP) and a fixed policy. Translation quality is reported using BLEU, chrF++, and COMET while latency is measured using LAAL and CAAL.
Text Translation
ASR Transcript
Latency
Speech Quality
System
BLEU ↑
chrF ↑
COMET ↑
BLEU ↑
chrF ↑
COMET ↑
CAAL ↓
LAAL ↓
UTMOS-V2 ↑
SpkSim ↑
Hibiki-Zero
32.9
59.9
75.8
31.4
58.2
74.9
1.39
2.11
2.1
0.47
SeamlessM4T
36.0
63.1
79.6
32.2
58.2
74.7
1.23
1.89
1.9
0.31
FAST-CAP
35.2
61.2
79.0
27.6
54.3
70.5
0.85
1.25
1.8
0.44
FAST-CAP-L
35.8
62.1
80.6
32.8
58.6
77.0
0.85
1.25
2.0
0.53
TABLE IV: Comparison of translation performance on the CVSS-T test set. Translation quality is evaluated with BLEU, chrF++, and COMET, with speech metrics computed on ASR transcriptions. Audio quality is measured using UTMOS-V2, and speaker similarity is measured using cosine similarity.
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab