How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems formulate SimulST as a multi-turn dialogue task, requiring specialized interleaved training data and relying on computationally expensive large language model (LLM) inference for decision-making. In this paper, we propose SimulSense, a novel framework for SimulST that mimics human interpreters by continuously reading input speech and triggering write decisions to produce translation when a new sense unit is perceived. Experiments against two state-of-the-art baseline systems demonstrate that our proposed method achieves a superior quality-latency tradeoff and substantially improved real-time efficiency, where its decision-making is up to 9.6x faster than the baselines.
Figures & tables
Figure 1: The Sense Units Detector (SUD) assigns a weight to each acoustic feature of the input stream. When the accumulated weight sum (including a prior residual weight r ) exceeds a triggering threshold γ , a sense unit is detected, which triggers the offline ST model to produce translations.
Figure 2: The overview of our SimulSense framework. Left: A Sense-Aware Transducer (SAT) training pipeline that guides a lightweight Sense Units Detector (SUD) model in learning human-interpreter-like sense unit segmentation. Right: The SUD model dynamically perceives sense units from acoustic features and drives the LLM-based offline ST model to perform simultaneous translation.
Figure 3: Quality-latency trade-off (latency in ms) compared to two state-of-the-art SimulST systems on the En → De, En → Ja, and En → Zh directions of the ACL 60/60 test set. SimulSense achieves a superior quality-latency tradeoff across all language pairs.
Model
Avg. Inference Time (ms)
RTF
NAIST-2025
371.3
0.180
Dialogue-LLM
116.2
0.130
Proposed SimulSense
38.6
0.016
Table 1: Efficiency comparison of the decision policies across different systems.
Figure 4: The impact of using different latency tags (latency in ms).
Model
Context Block
Layer
Heads
#Params (M)
WER (%)
SAT-Mini
×
—
—
41
74.5
SAT-Small
✓
2
2
100
67.7
SAT-Large
✓
8
4
232
69.1
Table 2: WER comparison across SAT variants with varying context block layers and attention heads.
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab