Learning Timeline-Preserving Speech-to-Speech Translation

Zhisheng Zheng; David Harwath

eess.ASOct 31, 2026

Learning Timeline-Preserving Speech-to-Speech Translation

Authors: Zhisheng Zheng, David Harwath

Organizations: The University of Texas at Austin

Abstract

Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at https://zhishengzheng.com/transdubber/.

Figures & tables

Figure 1: Illustration of timeline-preserving speech translation. Conventional S2ST may shift pauses or overrun the source endpoint; TransDubber instead preserves meaning while aligning translated speech with the source speaking windows, salient pauses, and utterance endpoint.

Figure 2: Stage 3 construction pipeline for timeline-preserving speech supervision.

Stage	Data type	# Samples	Src. h	Tgt. h	Target timing	Pause supervision
1	ASR text	928,661	3,014	–	Not applicable	Source-side markers only
1	AST text	883,563	2,854	–	Not applicable	None in Chinese target
2	Natural-pace S2ST	806,407	2,610	2,431	Unconstrained natural timing	No target-time labels
3	Timeline no-pause	412,945	687	687	Exact source duration	No selected pause spans
3	Timeline pause-aligned	229,331	682	682	Exact source duration	All selected pauses retained
Stage-summed total		3,260,907	9,847	3,800	–	–

Table 1: Retained training data by stage and supervision type. Source hours are summed across tasks and stages, not unique corpus hours. Stage 1 pause markers annotate source transcripts; only Stage 3 provides target-waveform timing supervision.

Figure 3: Autoregressive decoding in TransDubber . A pause-marked English transcript precedes Chinese text and speech-codec tokens, interleaved in blocks of five text and ten speech tokens. After text completion, speech generation continues until the final end-of-sequence token.

Panel of Figure 4 — Figure 4: Temporal rewards for Stage 4 reinforcement learning. (a) The duration reward is zero within ±2% of source duration and saturates at -4. (b) The pause reward penalizes missed or spurious silence, shifted pause centers, and endpoint mismatch.

	COMET $\uparrow$	MetricX-QE $\downarrow$	$\mathrm{Dur@}2\%\uparrow$	$\mathrm{Dur@}5\%\uparrow$	Pause F1 $\uparrow$	tIoU $\uparrow$	onMedAE $\downarrow$	Pauses / utt.
No intervention	0.788	4.12	61.5	92.2	0.550	0.322	0.162	0.89
Shift $+3$ tokens	0.784	4.24	60.7	91.5	0.476	0.278	0.174	0.88
Shuffle	0.786	4.17	58.8	90.2	0.369	0.227	0.174	0.81
Remove	0.795	3.91	58.3	89.1	0.231	0.156	0.192	0.44

Table 4: Pause-marker interventions in TransDubber . ASR inputs contain an average of 0.89 <pause> markers per utterance.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Criterion	Stage 2: natural pace	Stage 3: timeline preserving
Content realization	Chinese CER $\leq 0.10$	Chinese CER $\leq 0.10$
Script purity	No Latin letters in re-ASR	No Latin letters in intended text or re-ASR
Speaker consistency	Similarity $\geq 0.45$	Similarity $\geq 0.45$ per selected segment
Translation adequacy	No QE gate	MetricX-24 QE $\leq 5.0$
Duration	Target/source ratio in $[0.35,2.0]$	Exact source sample count

Appendix

Table 7: Acceptance criteria for synthesized training targets. Stage 3 speaker checks are applied during candidate selection.

Panel of Figure 5 — Figure 5: Training-data timing statistics. (a) Source-duration distributions, including short utterances added in Stage 3. (b) Target/source duration ratios, speaking rates, and supervised pause lengths. Stage 3 targets exactly match source duration, unlike Stage 2 natural-pace targets.

Hyperparameter	Stage 2	Stage 3	Stage 4
Peak learning rate	$10^{-4}$	$10^{-4}$	$3\times 10^{-6}$
Weight decay	0.02	0.02	0.01
LR decay schedule	Linear	Linear	Cosine
Optimization steps	20,000	12,000	100
Global batch per update	256 utterances	256 utterances	28 prompts
Codec-parameter LR multiplier	$10\times$	$10\times$	–

Appendix

Table 8: Training hyperparameters for natural-pace SFT (Stage 2), timeline-preserving SFT (Stage 3), and GSPO (Stage 4). Dashes indicate inapplicable settings.

Figure 6: Pause-alignment metrics on one shared timeline. Source $S$ and hypothesis $H$ pauses (all $\geq\tau_{\min}=0.4$ s) are compared at the same temporal origin. tIoU is the overlap of the silent spans; SER sums the missed silence $E_{\mathrm{miss}}$ , the inserted silence $E_{\mathrm{fa}}$ , and the endpoint mismatch $E_{\mathrm{dur}}$ , normalized by the source duration. F1@0.4 matches pause events one-to-one when their centers differ by at most 0.4 s ( $D$ is a false negative, $c$ a false positive); onsetMedAE@0.4 is the median absolute onset difference over the matched pairs. — Figure 6: Pause-alignment metrics on one shared timeline. Source $S$ and hypothesis $H$ pauses (all $\geq\tau_{\min}=0.4$ s) are compared at the same temporal origin. tIoU is the overlap of the silent spans; SER sums the missed silence $E_{\mathrm{miss}}$ , the inserted silence $E_{\mathrm{fa}}$ , and the endpoint mismatch $E_{\mathrm{dur}}$ , normalized by the source duration. [email protected] matches pause events one-to-one when their centers differ by at most 0.4 s ( $D$ is a false negative, $c$ a false positive); [email protected] is the median absolute onset difference over the matched pairs.

Figure 7: Human evaluation interface for audio naturalness and translation quality. Participants rate naturalness before viewing the Mandarin transcript, which is subsequently revealed for translation-quality assessment.

Panel of Figure 8 — Figure 8: Top-p and temperature decoding settings. Left axes show duration accuracy (Dur@2%); right axes show pause overlap (tIoU). Dashed lines mark the default settings.

Domain	System	COMET $\uparrow$	MetricX-QE $\downarrow$
CoVoST2	UniSS	0.738	5.68
CoVoST2	TransDubber	0.745	5.18
GigaSpeech	UniSS	0.732	5.64
GigaSpeech	TransDubber	0.809	3.61

Appendix

Table 9: Domain-specific translation quality on the ASR-mediated track. Unlike Table 2 , these scores are computed from ASR transcriptions of the generated speech and therefore also reflect speech realization errors.

Stage-3 supervision	COMET $\uparrow$	MetricX-QE $\downarrow$	Dur@2% $\uparrow$	Dur@5% $\uparrow$	Pause F1 $\uparrow$	tIoU $\uparrow$	SER $\downarrow$
$\mathcal{D}_{no{\text{-}}pause}$	0.793	3.77	33.4	63.9	0.180	0.134	0.258
$\mathcal{D}_{\mathrm{pause{\text{-}}aligned}}$	0.767	4.52	28.1	59.7	0.523	0.284	0.230
Full ( $\mathcal{D}_{no{\text{-}}pause}\cup\mathcal{D}_{\mathrm{pause{\text{-}}aligned}}$ )	0.772	4.38	48.6	83.3	0.518	0.293	0.198

Appendix

Table 10: Ablation of the Stage-3 supervision composition.

RL initialization	COMET $\uparrow$		$\mathrm{Dur@}2\%$ (%) $\uparrow$		tIoU $\uparrow$		Valid rollout rate
RL initialization	Pre-RL	Post-RL	Pre-RL	Post-RL	Pre-RL	Post-RL
Stage 1 checkpoint	0.823	N/A	–	N/A	–	N/A	0%
Stage 2 checkpoint	0.823	0.829	5.2	5.2	0.086	0.097	93.8%
Stage 3 checkpoint	0.772	0.788	48.6	61.5	0.293	0.322	100%

Appendix

Table 11: Reinforcement learning initialized from text grounding (Stage 1), natural-pace speech training (Stage 2), or timeline-preserving supervision (Stage 3).

System	Success (%) $\uparrow$	COMET $\uparrow$	MetricX-QE $\downarrow$	CER (%) $\downarrow$	$\mathrm{Dur@}10\%\uparrow$	Pause F1 $\uparrow$	tIoU $\uparrow$	SER $\downarrow$	onMedAE $\downarrow$
TransDubber	99.4	0.788	4.12	2.68	98.7	0.550	0.322	0.128	0.162
TransDubber translated text + F5-TTS	100.0	0.788	4.12	8.14	100.0	0.263	0.172	0.216	0.277
Stage-3 construction pipeline	47.5	0.724	4.88	5.07	100.0	0.642	0.481	0.081	0.067

Appendix

Table 13: Comparison with F5-TTS resynthesis of TransDubber translations and the Stage 3 construction pipeline.

Speech Translation

Speech translation converts spoken language into text or speech in another language, including real-time systems that balance fidelity and latency.

See all 62 papers

DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation
Sep 27, 2026Yayue Deng, Dingdong Wang, Yuxuan Hu +7Speech Translation Reinforcement Learning
The Chinese University of Hong Kong, China · Microsoft Corporation · Work done during an internship at Microsoft
X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
Oct 23, 2026Jeongsoo Choi, Ji-Hoon Kim, Junseok Ahn +1Speech Translation Audio-Visual Synchronization
KAIST · Chung-Ang University
SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
Jul 22, 2026Rongshen He, Xinyu Liang, Dekun Chen +3Speech Translation Simultaneous Speech Translation
1The Chinese University of Hong Kong, Shenzhen

Sep 27, 2026J/K move · Enter open · S save

DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation

Yayue Deng, Dingdong Wang, Yuxuan Hu, Jinyu Li+6

The Chinese University of Hong Kong, China · Microsoft Corporation · Work done during an internship at Microsoft

Speech Translation Reinforcement Learning Speech Language Models

Open paper AskPDF arXiv

Speech-to-speech translation (S2ST) in time-sensitive applications such as video dubbing requires not only semantic fidelity and speaker preservation, but also strict duration consistency to avoid audio-visual misalignment. However, existing S2ST systems largely generate target speech without explicit temporal planning, making duration control an unresolved challenge. We introduce DuraS2ST, a duration-aligned reasoning framework that enables a single speech language model to first generate an explicit chain-of-thought (CoT) for planning target wording and phonetic length, and then synthesize the corresponding speech tokens. To support this paradigm, we construct DuraSet-440K, a high-quality duration-aligned CoT corpus for supervised initialization. We further optimize the model with multi-modal multi-dimensional reinforcement learning, using a Duration Margin Reward to balance translation quality and duration consistency, and Modality-Aware Reward Attribution to assign rewards to appropriate token spans. Experiments on CVSS-T show that DuraS2ST achieves a strong balance between translation quality and duration consistency, outperforming competitive open-source and commercial baselines. Project page: https://github.com/Mia11939/DuraS2ST.

Abstract

Speech Translation

Explore similar work

DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation

X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision