Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation

Joseph Liu; Nameer Hirschkind; Xiao Yu; Mahesh Kumar Nandwana

cs.LGApr 10, 2026

Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation

Authors: Joseph Liu, Nameer Hirschkind, Xiao Yu, Mahesh Kumar Nandwana

Organizations: Roblox

Abstract

Simultaneous Speech Translation (SimulST) requires balancing high translation quality with low latency. Recent work introduced REINA, a method that trains a Read/Write policy based on estimating the information gain of reading more audio. However, we find that information-based policies often lack temporal context, leading the policy to bias itself toward reading most of the audio before starting to write. We improve REINA using two distinct strategies: a supervised alignment network (REINA-SAN) and a timestep-augmented network (REINA-TAN). Our results demonstrate that while both methods significantly outperform the baseline and resolve stability issues, REINA-TAN provides a slightly superior Pareto frontier for streaming efficiency, whereas REINA-SAN offers more robustness against 'read loops'. Applied to Whisper, both methods improve the pareto frontier of streaming efficiency as measured by Normalized Streaming Efficiency (NoSE) scores up to 7.1% over existing competitive baselines.

Figures & tables

Figure 1: Information Gain Given a fixed % of source audio, we plot information gain for each label token. There is a clearly visible Read - Write boundary at the token hull .

Figure 2: Read Loop Despite ingesting more and more audio over time, the policy continues to predict roughly the same value, corresponding to Read decisions and entering a Read Loop.

Panel of Figure 3 — Figure 3: Latency–quality trade-off of Whisper policy ablations. We plot XComet-XL versus LAAL by sweeping the policy threshold and averaging over de/fr/es $\rightarrow$ en. REINA-TAN achieves the strongest Pareto frontier across both datasets.

Figure 4: Mean per-token latency (actual emission time minus monotonic alignment time boundaries) as a function of relative token position within the sentence, with 95% confidence intervals for 2000 samples on the dev split of CVSS-C.

De $\rightarrow$ En
	FLEURS			Europarl
Model	LAAL(s)	BLEU	X-COMET	LAAL(s)	BLEU	X-COMET
REINA	1.33	25.2	.696	1.40	16.3	.608
REINA-All	1.21	26.9	.772	1.26	21.8	.760
REINA-SAN	1.03	28.6	.794	1.09	21.4	.755
REINA-TAN	1.03	27.9	.775	0.98	20.5	.721

Table 1: Streaming evaluation results on FLEURS and Europarl benchmarks. Operating points are selected to represent low and medium latency regimes across languages.

Metric / Model	de $\rightarrow$ en	fr $\rightarrow$ en	es $\rightarrow$ en
NoSE Bounds $[x,y]$ (s)	$[1.90,2.05]$	$[1.67,1.79]$	$[1.78,1.92]$
Whisper-Large v3 Non-Streaming (BLEU)	33.40	31.02	22.70
SeamlessM4Tv2 (BLEU)	37.00	33.97	25.49
SeamlessStreaming (NoSE)	0.925	0.940	0.936
Whisper-REINA (NoSE)	0.921	0.955	0.944
Whisper-REINA-SAN (NoSE)	0.987	0.984	0.951

Table 2: Comparison of Normalized Streaming Efficiency (NoSE) and non-streaming baselines on FLEURS. NoSE values are calculated within the specified latency boundaries

[x,y]

Simultaneous Speech Translation

Simultaneous speech translation converts speech into another language as audio arrives, balancing translation quality, latency, and fluent delivery.

See all 26 papers

Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation
Sep 25, 2026Filip Tăşădan, Ema Tomanová, Ondrej Lopuch +2Simultaneous Speech Translation Transformer Attention
University of Copenhagen
Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
Apr 16, 2025Biao Fu, Donglei Yu, Minpeng Liao +5Speech Translation Simultaneous Speech Translation
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab
SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning
Oct 26, 2026Haotian Tan, Hiroki Ouchi, Sakriani SaktiSimultaneous Speech Translation Speech Language Models
Nara Institute of Science and Technology, Japan

Sep 25, 2026J/K move · Enter open · S save

Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation

Filip Tăşădan, Ema Tomanová, Ondrej Lopuch, Paweł Bilko+1

University of Copenhagen

Simultaneous Speech Translation Transformer Attention

Open paper AskPDF arXiv

Simultaneous speech-to-text translation (Simul-S2TT) consists of generating partial translations while the incoming audio frames are processed by the system. However, the streaming nature of this setup creates the challenge of deciding the best moment to perform an accurate translation while minimizing the delay. To address this challenge, we utilize the cross-attention mechanism of the encoder-decoder architecture to find the right alignment between the input speech frames and the target text tokens. In this paper, we propose the Recent Frame Attention Policy (RFAP) and the Dual-Condition Attention Policy (DCAP) that allow offline trained speech-to-text translation models to be used in streaming scenarios without requiring additional training. Results on three different language translation pairs over the CVSS-C corpus show that the RFAP is able to surpass other policies with gains of up to 4.0 BLEU while reducing the translation delay by almost 1 second. Moreover, the DCAP is able to preserve a high translation quality when the latency is very low.

Abstract

Simultaneous Speech Translation

Explore similar work

Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation

Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture

SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning