Most of modern neural machine translation (NMT) models are based on an encoder-decoder framework with an attention mechanism. While they perform well on standard datasets, they can have trouble in translation of long inputs that are rare or unseen during training. Incorporating target syntax is one approach to dealing with such length-related problems. We propose a novel syntactic decoder that generates a target-language dependency tree in a top-down, left-to-right order. Experiments show that the proposed top-down string-to-tree decoding generalizes better than conventional sequence-to-sequence decoding in translating long inputs that are not observed in the training data.
Figures & tables
Figure 1: Basic unit of our model that generates a node and a subtree that belongs to it. Each triangle in the figure represents a subtree.
Figure 2: An example dependency tree generated by the procedure in Fig. 3 . Most empty symbols are omitted for brevity.
input
output
stack
1
ROOT
VBZ
L∣W∣R
2
LEFT::VBZ
IN
L∣W∣R∣S∣W∣R
3
LEFT::IN
ϕL
W∣R∣S∣W∣R
4
WORD:: ϕL
Until
W∣R∣S∣W∣R
5
WORD::Until
ϕW
R∣S∣W∣R
6
RIGHT:: ϕW
RB
L∣W∣R∣S∣S∣W∣R
Figure 3: An example action sequence that generates the tree in Fig. 2 . “::” stands for a concatenation operation. L,W,S and R stand for LEFT, WORD, SIBLING and RIGHT actions, respectively.
Figure 4: Illustration of the proposed decoder. Dotted lines represent hidden states.
Topdown
Transformer
dev
35.78
35.76
test
34.17
33.93
Table 1: Results on WMT16 Ro-En dataset.
Num. verbs
Topdown
Transformer
0
39.58 ± 0.49
40.00 ± 1.48
1
39.42 ± 1.02
38.56 ± 0.66
2
38.76 ± 0.64
38.62 ± 0.59
3
37.62 ± 0.78
37.39 ± 0.73
4
37.15 ± 0.57
37.11 ± 0.72
5
36.16 ± 0.79
35.94 ± 0.95
Table 2: Results on Tilde-subset based on number of verbs in the source sentence, averaged over five test sets with different random seeds.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
We present an ensemble-driven self-training framework for unsupervised neural machine translation (UNMT). Starting from a primary language pair, we train multiple UNMT models that share the same translation task but differ in an auxiliary language, inducing structured diversity across models. We then generate pseudo-translations for the primary pair using token-level ensemble decoding, averaging model predictions in both directions. These ensemble outputs are used as synthetic parallel data to further train each model, allowing the models to improve via shared supervision. At deployment time, we select a single model by validation performance, preserving single-model inference cost. Experiments show statistically significant improvements over single-model UNMT baselines, with mean gains of 1.7 chrF when translating from English and 0.67 chrF when translating into English.
Low-resource machine translation requires methods that differ from those used for high-resource languages. This paper proposes a novel in-context learning approach to support low-resource machine translation of the Coptic language to English, with syntactic augmentation from Universal Dependencies parses of input sentences. Building on existing work using bilingual dictionaries to support inference for vocabulary items, we add several representations of syntactic analyses to our inputs , specifically exploring the inclusion of raw parser outputs, verbalizations of parses in plain English, and targeted instructions of difficult constructions identified in sub-trees and how they can be translated. Our results show that while syntactic information alone is not as useful as dictionary-based glosses, combining retrieved dictionary items with syntactic information achieves significant gains across model sizes, achieving new state-of-the-art translation results for Coptic.
Abhishek Purushothama, Emma Thronson, Alexia Guo +1
Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.
Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov +2
Saarland University · University of Göttingen · German Research Center for Artificial Intelligence (DFKI) +1