Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at https://zhishengzheng.com/transdubber/.
Figures & tables
Figure 1: Illustration of timeline-preserving speech translation. Conventional S2ST may shift pauses or overrun the source endpoint; TransDubber instead preserves meaning while aligning translated speech with the source speaking windows, salient pauses, and utterance endpoint.
Figure 2: Stage 3 construction pipeline for timeline-preserving speech supervision.
Stage
Data type
# Samples
Src. h
Tgt. h
Target timing
Pause supervision
1
ASR text
928,661
3,014
–
Not applicable
Source-side markers only
1
AST text
883,563
2,854
–
Not applicable
None in Chinese target
2
Natural-pace S2ST
806,407
2,610
2,431
Unconstrained natural timing
No target-time labels
3
Timeline no-pause
412,945
687
687
Exact source duration
No selected pause spans
3
Timeline pause-aligned
229,331
682
682
Exact source duration
All selected pauses retained
Stage-summed total
3,260,907
9,847
3,800
–
–
Table 1: Retained training data by stage and supervision type. Source hours are summed across tasks and stages, not unique corpus hours. Stage 1 pause markers annotate source transcripts; only Stage 3 provides target-waveform timing supervision.
Figure 3: Autoregressive decoding in TransDubber . A pause-marked English transcript precedes Chinese text and speech-codec tokens, interleaved in blocks of five text and ten speech tokens. After text completion, speech generation continues until the final end-of-sequence token.
Figure 4: Temporal rewards for Stage 4 reinforcement learning. (a) The duration reward is zero within ±2% of source duration and saturates at -4. (b) The pause reward penalizes missed or spurious silence, shifted pause centers, and endpoint mismatch.
COMET ↑
MetricX-QE ↓
Dur@2%↑
Dur@5%↑
Pause F1 ↑
tIoU ↑
onMedAE ↓
Pauses / utt.
No intervention
0.788
4.12
61.5
92.2
0.550
0.322
0.162
0.89
Shift +3 tokens
0.784
4.24
60.7
91.5
0.476
0.278
0.174
0.88
Shuffle
0.786
4.17
58.8
90.2
0.369
0.227
0.174
0.81
Remove
0.795
3.91
58.3
89.1
0.231
0.156
0.192
0.44
Table 4: Pause-marker interventions in TransDubber . ASR inputs contain an average of 0.89 <pause> markers per utterance.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Stage 2: natural pace
Stage 3: timeline preserving
Content realization
Chinese CER ≤0.10
Chinese CER ≤0.10
Script purity
No Latin letters in re-ASR
No Latin letters in intended text or re-ASR
Speaker consistency
Similarity ≥0.45
Similarity ≥0.45 per selected segment
Translation adequacy
No QE gate
MetricX-24 QE ≤5.0
Duration
Target/source ratio in [0.35,2.0]
Exact source sample count
Appendix
Table 7: Acceptance criteria for synthesized training targets. Stage 3 speaker checks are applied during candidate selection.
Figure 5: Training-data timing statistics. (a) Source-duration distributions, including short utterances added in Stage 3. (b) Target/source duration ratios, speaking rates, and supervised pause lengths. Stage 3 targets exactly match source duration, unlike Stage 2 natural-pace targets.
Hyperparameter
Stage 2
Stage 3
Stage 4
Peak learning rate
10−4
10−4
3×10−6
Weight decay
0.02
0.02
0.01
LR decay schedule
Linear
Linear
Cosine
Optimization steps
20,000
12,000
100
Global batch per update
256 utterances
256 utterances
28 prompts
Codec-parameter LR multiplier
10×
10×
–
Appendix
Table 8: Training hyperparameters for natural-pace SFT (Stage 2), timeline-preserving SFT (Stage 3), and GSPO (Stage 4). Dashes indicate inapplicable settings.
Figure 6: Pause-alignment metrics on one shared timeline. Source S and hypothesis H pauses (all ≥τmin=0.4 s) are compared at the same temporal origin. tIoU is the overlap of the silent spans; SER sums the missed silence Emiss , the inserted silence Efa , and the endpoint mismatch Edur , normalized by the source duration. [email protected] matches pause events one-to-one when their centers differ by at most 0.4 s ( D is a false negative, c a false positive); [email protected] is the median absolute onset difference over the matched pairs.
Figure 7: Human evaluation interface for audio naturalness and translation quality. Participants rate naturalness before viewing the Mandarin transcript, which is subsequently revealed for translation-quality assessment.
Figure 8: Top-p and temperature decoding settings. Left axes show duration accuracy (Dur@2%); right axes show pause overlap (tIoU). Dashed lines mark the default settings.
Domain
System
COMET ↑
MetricX-QE ↓
CoVoST2
UniSS
0.738
5.68
TransDubber
0.745
5.18
GigaSpeech
UniSS
0.732
5.64
TransDubber
0.809
3.61
Appendix
Table 9: Domain-specific translation quality on the ASR-mediated track. Unlike Table 2 , these scores are computed from ASR transcriptions of the generated speech and therefore also reflect speech realization errors.
Stage-3 supervision
COMET ↑
MetricX-QE ↓
Dur@2% ↑
Dur@5% ↑
Pause F1 ↑
tIoU ↑
SER ↓
Dno-pause
0.793
3.77
33.4
63.9
0.180
0.134
0.258
Dpause-aligned
0.767
4.52
28.1
59.7
0.523
0.284
0.230
Full ( Dno-pause∪Dpause-aligned )
0.772
4.38
48.6
83.3
0.518
0.293
0.198
Appendix
Table 10: Ablation of the Stage-3 supervision composition.
RL initialization
COMET ↑
Dur@2% (%) ↑
tIoU ↑
Valid rollout rate
Pre-RL
Post-RL
Pre-RL
Post-RL
Pre-RL
Post-RL
Stage 1 checkpoint
0.823
N/A
–
N/A
–
N/A
0%
Stage 2 checkpoint
0.823
0.829
5.2
5.2
0.086
0.097
93.8%
Stage 3 checkpoint
0.772
0.788
48.6
61.5
0.293
0.322
100%
Appendix
Table 11: Reinforcement learning initialized from text grounding (Stage 1), natural-pace speech training (Stage 2), or timeline-preserving supervision (Stage 3).
System
Success (%) ↑
COMET ↑
MetricX-QE ↓
CER (%) ↓
Dur@10%↑
Pause F1 ↑
tIoU ↑
SER ↓
onMedAE ↓
TransDubber
99.4
0.788
4.12
2.68
98.7
0.550
0.322
0.128
0.162
TransDubber translated text + F5-TTS
100.0
0.788
4.12
8.14
100.0
0.263
0.172
0.216
0.277
Stage-3 construction pipeline
47.5
0.724
4.88
5.07
100.0
0.642
0.481
0.081
0.067
Appendix
Table 13: Comparison with F5-TTS resynthesis of TransDubber translations and the Stage 3 construction pipeline.
Speech-to-speech translation (S2ST) in time-sensitive applications such as video dubbing requires not only semantic fidelity and speaker preservation, but also strict duration consistency to avoid audio-visual misalignment. However, existing S2ST systems largely generate target speech without explicit temporal planning, making duration control an unresolved challenge. We introduce DuraS2ST, a duration-aligned reasoning framework that enables a single speech language model to first generate an explicit chain-of-thought (CoT) for planning target wording and phonetic length, and then synthesize the corresponding speech tokens. To support this paradigm, we construct DuraSet-440K, a high-quality duration-aligned CoT corpus for supervised initialization. We further optimize the model with multi-modal multi-dimensional reinforcement learning, using a Duration Margin Reward to balance translation quality and duration consistency, and Modality-Aware Reward Attribution to assign rewards to appropriate token spans. Experiments on CVSS-T show that DuraS2ST achieves a strong balance between translation quality and duration consistency, outperforming competitive open-source and commercial baselines. Project page: https://github.com/Mia11939/DuraS2ST.
Yayue Deng, Dingdong Wang, Yuxuan Hu +7
The Chinese University of Hong Kong, China · Microsoft Corporation · Work done during an internship at Microsoft
Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task is to produce a faithful translation that is naturally speakable within the source timing, which is non-trivial because languages differ in duration, rhythm, and articulation. In this paper, we propose X-Dubbing, which formulates the task as a joint problem of time-aware speech-to-text translation and video-aware text-to-speech. Specifically, our method consists of a Dubbing Translator that generates temporally structured scripts and a Voice Actor that renders them in synchrony with the source video, both grounded in a language-agnostic timing budget. We further refine the translator with dubbing-aware feedback from the synthesized speech, encouraging scripts that yield well-synchronized dubbing. This design requires no paired dubbing data, as the two components are trained on speech-text and monolingual audio-visual data respectively. Extensive experiments show that X-Dubbing outperforms recent baselines across diverse metrics, including isochrony and audio-visual correspondence.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only ∼2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.