Scaling Unsupervised Word Alignment to Documents via Structural Constraints

Michelle Wastl; Jannis Vamvas; Rico Sennrich

cs.CLAug 21, 2026

Scaling Unsupervised Word Alignment to Documents via Structural Constraints

Authors: Michelle Wastl, Jannis Vamvas, Rico Sennrich

Organizations: Department of Computational Linguistics, University of Zurich

Abstract

Word alignment has traditionally been studied between sentences, but many cross-lingual tasks increasingly require correspondences across full documents. While recent multilingual embedding models can encode long inputs, we show that applying algorithms designed for sentences directly to documents leads to performance degradation. To address this, we introduce CTFAlign, a lightweight, training-free approach for document-level word alignment. CTFAlign applies a coarse-to-fine refinement strategy that restricts the alignment search space to semantically similar regions. Additionally, we introduce MDPAlign, a simpler alternative that constrains alignments by position with a main diagonal prior. Both approaches operate directly on full documents without relying on sentence segmentation or sentence alignment. We evaluate these methods across six language pairs varying in typological distance, resourcedness, and document length. Averaged over three models, CTFAlign reduces word alignment error rate from 0.412 to 0.326. These gains transfer downstream, leading to improvements in document-level translation coverage evaluation and recognition of semantic differences. We release CTFAlign as a Python package and make the code and data to reproduce our experiments publicly available.

Figures & tables

Panel of Figure 1 — Figure 1: Performing word alignment on a pair of English and Japanese Wikipedia articles shows the differences between alignment algorithms and the correct alignments , false negatives , and false positives they predict. SimAlign Jalili Sabet et al. (2020) produces some stray false positives that are implausible given the structure of the documents. MDPAlign focuses on the diagonal but misses the bend towards the end; CTFAlign is closest to the gold alignment. The figure is based on Qwen3-Embedding-4B embeddings and subsamples 10% of points for visualization. The two axes are the token positions in the document pair.

Panel of Figure 2 — Figure 2: Illustration of the two MDPAlign variants. Both assume similar relative positions of corresponding words across documents and constrain alignment search toward the main diagonal. Alignments shown in search space with darkness of gray indicating suppression of off-diagonal similarities.

Panel of Figure 3 — Figure 3: Illustration of one iteration of the CTFAlign algorithm. White regions denote searchable grid space, while gray regions are masked because they fall outside the aligned coarse blocks identified at the previous resolution. The process repeats from an initial $2\times 2$ grid down to the token-level $m\times n$ grid.

Figure 4: AER (lower is better) on the test set with Qwen3-Embedding-4B , mmBERT-base , and LaBSE . Different constrained word alignment approaches are denoted by different colors, the document-level baseline is shown in gray, and the dashed lines indicate sentence-level SimAlign performance of the given model.

Granularity	en-fr	en-ro	en-ja	en-zh	la-gr	en-cz
Sentences	0.185	0.437	0.628	0.399	0.179	0.344
Documents	0.995	0.968	0.986	0.935	0.996	0.982

Table 1: GPT-5.4-mini word alignment AER on the test set.

Segment-level
	ja-zh		en-de
Method	Om.	Add.	Om.	Add.
DiffAlign	0.758	0.769	0.856	0.877
DiffAlign MDP+Argmax	0.789	0.783	0.867	0.885
DiffAlign CTF+Argmax	0.801	0.779	0.869	0.882
DiffAlign CTF+Itermax	0.781	0.776	0.862	0.880

Table 2: Token-level ROC AUC for translation coverage evaluation with Qwen3-Embedding-4B .

Method	en–de	en–fr	en–it
DiffAlign	0.286	0.156	0.318
DiffAlign MDP+Argmax	0.312	0.158	0.312
DiffAlign CTF+Argmax	0.330	0.189	0.353
DiffAlign CTF+Itermax	0.319	0.172	0.344

Table 3: Test set Spearman

\rho

for SwissGov-RSD with Qwen3-Embedding-4B .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Lang	Docs		Sents		Mean len
	dev	test	dev	test	doc	sent
en-fr	1	1	37	447	22408	92
en-ro	1	10	17	248	2901	119
en-ja	8	7	653	582	5516	66
en-zh	47	52	450	450	646	70
la-gr	1	1	25	75	5837	116

Appendix

Table 4: Word alignment dataset statistics. Lengths are target-side ( text_b ) character counts; only means shown for space. -- indicates no data for that split.

Lang	Docs		Sents		Mean len		Err %
	tot.	err.	tot.	err.	doc	sent	add.	omi.
en-de	3230	284	9232	320	724	253	0.3	0.2
ja-zh	2940	554	8385	617	264	92	1.0	1.1

Appendix

Table 5: WMT 2024 MQM dataset statistics. Doc/sent lengths are target-side character counts; only means shown. err. columns count documents/sentences containing at least one omission or addition. add. and omi. report the fraction (%) of non-punctuation characters labeled as additions or omissions.

Panel of Figure 6 — Figure 6: 1-shot prompt example used for the LLM baseline based on the “full mode” prompt introduced by Miao et al. (2025) .

Sentence-level
Model / Method	P	R	F1	AER
XLM-R-large
SimAlign + Argmax	0.777	0.635	0.692	0.308
SimAlign + Itermax	0.689	0.692	0.685	0.317
mBERT
SimAlign + Argmax	0.816	0.625	0.701	0.297

Appendix

Table 6: Macro-average pilot word alignment development results across language pairs. Best sentence-level results are underlined and best document-level results are bolded . For AER lower is better.

Figure 7: F1 word alignment results across layers for different language pairs with Argmax on the development set for both granularities for Qwen3-Embedding-4B , mmBERT-base , and LaBSE-concatenated-sentences .

Figure 8: AER (lower is better) and F1 (higher is better) results for different

k

w

values for Qwen3-Embedding-4B , mmBERT-base , and LaBSE-concatenated-sentences .

Last layer
Method	en–de	en–fr	en–it
DiffAlign	0.186	0.098	0.206
DiffAlign MDP+Argmax	0.227	0.110	0.228
DiffAlign CTF+Itermax	0.262	0.140	0.254
Layer 20
DiffAlign	0.286	0.156	0.318

Appendix

Table 7: SwissGov-RSD development set Spearman

\rho

for DiffAlign variants using Qwen3-Embedding-4B across en–de, en–fr, and en–it. Bold indicates the best result per language pair.

Approach	$k$ / $w$	F1	AER
CTFAlign	0	0.140	0.858
	2	0.626	0.369
	4	0.666	0.331
	6	0.673	0.325
	8	0.677	0.321
	10	0.677	0.322

Appendix

Table 8: Average dev-set F1 and AER (lower is better) across LaBSE , mmBERT , and Qwen for each hyperparameter value. Approaches were combined with Argmax. Best per approach and column in bold .

Model	en-fr				en-ro				en-ja				en-zh				la-gr				Avg
	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER
Sentence-level
GPT-5.4-mini	0.820	0.799	0.809	0.187	0.632	0.592	0.611	0.389	0.255	0.576	0.354	0.646	0.547	0.669	0.602	0.398	0.762	0.849	0.803	0.198	0.603	0.697	0.636	0.364
LaBSE
SimAlign + Argmax	0.966	0.950	0.958	0.040	0.971	0.743	0.842	0.158	0.761	0.444	0.561	0.439	0.892	0.834	0.862	0.138	0.702	0.664	0.682	0.317	0.858	0.727	0.781	0.218
SimAlign + Itermax	0.886	0.979	0.930	0.083	0.874	0.834	0.854	0.146	0.652	0.532	0.586	0.414	0.777	0.896	0.832	0.168	0.604	0.753	0.670	0.331	0.759	0.799	0.774	0.228

Appendix

Table 9: Word alignment development results (P, R, F1, AER). Fixed bandwidths: MDPAlign strict

k{=}50

, MDPAlign fuzzy

k{=}150

, CTFAlign

w{=}8

. Sentence-level rows are upper-bound baselines; best per column underlined . Document-level best per column bold .

	en-fr				en-ro				en-ja				en-zh				la-gr				en-cz				Avg
Model	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER	P	R	F1	AER
Sentence-level
LaBSE SimAlign + Argmax	0.969	0.948	0.958	0.040	0.921	0.676	0.780	0.220	0.755	0.456	0.569	0.431	0.880	0.819	0.848	0.151	0.710	0.707	0.708	0.292	0.883	0.899	0.891	0.109	0.853	0.751	0.792	0.207
mmBERT SimAlign + Argmax	0.958	0.932	0.945	0.053	0.906	0.639	0.749	0.251	0.738	0.419	0.535	0.465	0.867	0.757	0.808	0.192	0.690	0.711	0.700	0.300	0.878	0.856	0.867	0.133	0.840	0.719	0.767	0.232
Qwen SimAlign + Argmax	0.932	0.826	0.876	0.117	0.795	0.527	0.634	0.366	0.647	0.354	0.458	0.542	0.843	0.670	0.747	0.253	0.646	0.774	0.704	0.297	0.809	0.796	0.802	0.197	0.779	0.658	0.704	0.295
GPT-5.4-mini 1-shot prompt	0.831	0.787	0.808	0.185	0.552	0.574	0.563	0.437	0.270	0.599	0.372	0.628	0.543	0.674	0.601	0.399	0.788	0.861	0.823	0.179	0.638	0.678	0.657	0.344	0.604	0.696	0.637	0.362

Appendix

Table 10: Detailed word alignment test set results (P, R, F1, AER) across language pairs and models. Sentence-level rows are upper-bound baselines; best per column underlined . Document-level best per column bold .

Panel of Figure 9 — Figure 9: (a) CTFAlign + Itermax with mmBERT failure cases on en–fr in comparison to partially recovered alternative (b) CTFAlign + Argmax. Colors denote correct alignments , false negatives , false positives .

OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment
Aug 19, 2026Mengpeng Yang, Jingxu Yang, Chao Chen +3Cross-Lingual Representation Alignment
WPS Qingqiu / Wuhan, China
STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation
Aug 27, 2026Yichen Dong, Hao Wang, Junhui Li +3Machine Translation Document-Level MT
School of Computer Science and Technology, Soochow University, Suzhou, China · Alibaba Group, Hangzhou, China
Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders
Mar 19, 2026Yana Veitsman, Yihong Liu, Hinrich SchützeCross-Lingual Representation Alignment Cross-Lingual Transfer
University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)

Aug 19, 2026J/K move · Enter open · S save

OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment

Mengpeng Yang, Jingxu Yang, Chao Chen, Tian Xia+2

WPS Qingqiu / Wuhan, China

Cross-Lingual Representation Alignment

Open paper AskPDF arXiv

Cross-lingual sequence alignment is fundamental for building and exploiting parallel corpora, spanning mappings from documents and sentences down to words and subwords. Existing tools, however, typically specialize in a single granularity, so practitioners often need separate systems for word- and sentence-level alignment---especially in multilingual and long-text settings. We present OmniAlign, a unified multilingual aligner that supports both word-level and sentence-level alignment with a single lightweight model. Built on an encoder-only backbone with strong long-context modeling, OmniAlign induces word alignments from contextualized token similarity matrices, and obtains document-level

m

n

sentence alignments via sentence embeddings combined with dynamic programming. To balance fine-grained alignment accuracy and sentence-representation quality, we use a four-stage training pipeline: alignment-oriented continued pre-training, self-supervised learning, supervised fine-tuning on human annotations, and sentence-embedding distillation from a strong multilingual teacher. Experiments show that OmniAlign achieves highly competitive performance on both word- and sentence-alignment benchmarks and generalizes well to unseen language pairs. Surprisingly, later-stage supervised fine-tuning on short texts further improves alignment quality while retaining the long-context understanding acquired in earlier training, keeping the model robust on long-text word alignment. \normalsize {\color{blue}Code: https://github.com/MilkDargon/OmniAlign}\par {\color{blue}Model: https://huggingface.co/WPS-Qingqiu/OmniAlign}

Abstract

Explore similar work

OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment

STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders