Large Language Models (LLMs) have shown strong performance in low-resource offline translation; however, extending them to simultaneous speech-to-speech translation (Simul-S2ST) remains challenging due to the scarcity of causally aligned training data with high cross-lingual speaker fidelity. In addition, existing approaches rely on fixed translation policy or confidence heuristics, leading to suboptimal quality and higher latency. We propose a causality-aware Simul-S2ST framework with a novel data pipeline that generates high-fidelity, causally aligned segments with improved voice transfer. The framework introduces (i) a factorized S2ST architecture (FAST), (ii) a causality-aware adaptive policy (CAP), and (iii) causality-aware latency metric. Experiments on CVSS Spanish, German, and French show that FAST-CAP consistently improves the quality-latency trade-off, achieving up to +1.2 BLEU and a 26% relative latency reduction over a fixed policy. Despite using substantially less training data than existing systems, FAST-CAP achieves state-of-the-art results in speech translation quality and speaker fidelity while yielding up to a 38.8% relative reduction in latency.
Figures & tables
Es-En
De-En
Fr-En
UTMOS
Spk-Sim
UTMOS
Spk-Sim
UTMOS
Spk-Sim
CVSS-T (public)
2.4
0.22
2.5
0.26
2.5
0.25
CVSS-T (ours)
2.4
0.64
2.5
0.61
2.5
0.57
TABLE I: Comparison of speaker similarity and audio quality on the CVSS-T Dataset, with audio quality measured by UTMOS-V2.
Fig. 1: Overview of the data generation pipeline. The process includes (1) alignments generation, (2) unique mapping and causal pivot extraction, and (3) causality-aware adaptive chunks construction.
Fig. 2: Overview of the proposed FAST architecture with multistream joint sequence modeling.
Fig. 3: Illustration of LAAL and CAAL computation. Oracle delays (green) and system delays (blue) are paired according to the alignment mapping (arrows), and lagging values (red) are computed as the difference between each system delay and its aligned oracle delay.
Text Translation
ASR Transcript
#
System
LLM
Align.
BLEU
chrF
COMET
BLEU
chrF
COMET
1
FAST (IL)
Qwen
NFA
18.7
49.0
68.6
12.5
39.5
61.1
2
FAST (IL)
Qwen
MFA
23.1
49.7
70.0
13.5
40.2
62.5
3
FAST-Lat (MS)
Qwen
MFA
26.6
54.8
75.4
4.5
23.3
56.0
4
FAST (MS)
Qwen
MFA
26.7
54.6
75.5
20.8
48.6
68.2
5
FAST (MS)
Llama
MFA
25.1
54.3
75.4
19.9
48.0
67.5
TABLE II: Ablation study on the CVSS-T development set comparing alignment methods (NFA vs. MFA), multimodal representations (interleaved, IL, vs. multistream, MS), and TTS conditioning (text vs. LLM latent representations, Lat). Translation quality is evaluated with BLEU, chrF++, and COMET, with speech metrics computed on ASR transcriptions.
Avg.
Latency (s)
Translation Quality
System
input (s)
CAAL ↓
LAAL ↓
BLEU ↑
chrF ↑
COMET ↑
FAST-CAP (Sp-En)
0.4
0.33
0.57
12.1
47.0
67.1
FAST-CAP (Sp-En)
0.9
0.52
0.88
24.0
55.3
75.0
FAST-CAP (Sp-En)
1.5
0.80
1.15
27.8
57.6
78.0
FAST-Fixed (Sp-En)
2.0
0.87
1.17
26.7
54.6
75.5
FAST-CAP (De-En)
1.5
0.68
1.20
34.6
62.0
78.1
TABLE III: Comparison of FAST on CVSS-T development set using the causality-aware adaptive policy (CAP) and a fixed policy. Translation quality is reported using BLEU, chrF++, and COMET while latency is measured using LAAL and CAAL.
Text Translation
ASR Transcript
Latency
Speech Quality
System
BLEU ↑
chrF ↑
COMET ↑
BLEU ↑
chrF ↑
COMET ↑
CAAL ↓
LAAL ↓
UTMOS-V2 ↑
SpkSim ↑
Hibiki-Zero
32.9
59.9
75.8
31.4
58.2
74.9
1.39
2.11
2.1
0.47
SeamlessM4T
36.0
63.1
79.6
32.2
58.2
74.7
1.23
1.89
1.9
0.31
FAST-CAP
35.2
61.2
79.0
27.6
54.3
70.5
0.85
1.25
1.8
0.44
FAST-CAP-L
35.8
62.1
80.6
32.8
58.6
77.0
0.85
1.25
2.0
0.53
TABLE IV: Comparison of translation performance on the CVSS-T test set. Translation quality is evaluated with BLEU, chrF++, and COMET, with speech metrics computed on ASR transcriptions. Audio quality is measured using UTMOS-V2, and speaker similarity is measured using cosine similarity.
Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have shown strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on both in-domain (MuST-C) and out-of-domain (Europarl-ST) En-De and En-Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.
Biao Fu, Donglei Yu, Minpeng Liao +5
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
Simultaneous speech translation (SimulST) requires incremental translation under strict latency constraints, yet remains challenging for decoder-only LLM systems due to limited context and cross-lingual reordering. Recent approaches often introduce architectural changes or explicit read/write policies to control output timing, which can be brittle in conversational speech where segmentation boundaries are ambiguous. We present a simple data-driven alternative: fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, and teacher-labeled prefix-to-prefix (P2P) targets with bounded waiting for fine-tuning, yielding CSSEL-P2P, where CSSEL is our proposed chunked streaming speech encoder LLM. In our in-house conversational speech evaluation, CSSEL-P2P improves streaming quality by +1.54 COMETKiwi over the CSSEL streaming baseline at comparable latency (+0.15s Average Lagging), suggesting effective SimulST without architectural changes via P2P supervision.
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.