STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation

Yichen Dong; Hao Wang; Junhui Li; Linlong Xu; Longyue Wang; Weihua Luo

cs.CLAug 27, 2026

STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation

Authors: Yichen Dong, Hao Wang, Junhui Li, Linlong Xu, Longyue Wang, Weihua Luo

Organizations: School of Computer Science and Technology, Soochow University, Suzhou, China · Alibaba Group, Hangzhou, China

Abstract

Large Language Models (LLMs) have enabled a shift from sentence-level to document-to-document (Doc2Doc) machine translation, promising improved global coherence. However, document-to-document generation in a single pass frequently suffers from structural misalignment, manifesting as sentence omissions or hallucinations that violate the core requirement of source-target correspondence. To address this, we introduce Sentence Translation Alignment Rate (STAR), an auxiliary metric that explicitly quantifies sentence-level structural fidelity. Building on this, we propose STAR-masked Preference Optimization (StarPO), a framework that ranks document-level hypotheses by structural quality and utilizes a dynamic alignment mask to focus optimization on misaligned segments. Experimental results across news and literary domains demonstrate that StarPO significantly enhances translation quality and structural integrity. Notably, StarPO allows compact models to surpass the performance of massive proprietary systems like GPT-4o while maintaining superior token efficiency.

Figures & tables

\Block 2-1 System	Ideal	Structural Deviations
	1-to-1	1-to-0	0-to-1	Other
\Block [l]1-5 Doc2Doc translation in a single pass
LLaMA-3.1-8B	92.59	2.08	1.17	4.16
Qwen-2.5-7B	95.35	1.91	1.31	1.43
Qwen-3-4B	94.72	0.72	0.33	4.23
Deepseek-R1	95.03	4.85	0.03	0.09

Table 1: Alignment distribution ( in % ) for Zh

\Rightarrow

En on News-Commentary, computed by Gemini-2.5-Flash.

Figure 1: Overview of the proposed framework. (a) Sentence Translation Alignment Rate (STAR): Source and target documents are segmented and aligned to compute STAR, defined as the ratio of strict 1-to-1 alignments. (b) Preference Data Generation: Translation candidates are sampled and ranked by their STAR scores. Pairs exhibiting a score disparity larger than $\tau$ are selected as chosen ( $y_{w}$ ) and rejected ( $y_{l}$ ) samples. (c) STAR-Masked Preference Optimization (StarPO): A sentence-level mask is applied to the CPO objective, excluding well-aligned (1-to-1) sentences to focus optimization exclusively on structurally problematic segments.

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ Zh		\Block [c]1-2 Ru $\Leftrightarrow$ En		\Block [c]1-2 En $\Leftrightarrow$ Es		\Block [l]2-1 Avg.
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-12 Llama-3.1-8B-Instruct
Base	72.23	74.89	71.60	80.13	72.49	58.80	80.65	80.75	80.53	78.62	75.07
+ SFT	76.81	80.01	83.79	81.47	75.42	75.41	82.37	80.69	80.66	83.49	80.01
+ CPO	81.10	79.94	83.98	82.42	75.50	75.05	82.51	81.43	81.45	84.60	80.80
+ StarPO	81.55	80.11	84.05	82.50	76.33	77.08	82.53	81.71	82.03	84.91	81.28

Table 2: Performance in dCOMET score on the News-Commentary test set. Bold scores represent the global best performance and underlined scores represent the global second-best performance. Blue text background indicates that the improvement over the origin Base model achieves at least 85% accuracy with the human judgment Xu et al. (2024) ; Kocmi et al. (2024b) . Specifically, the improvement needs a minimin of

\geq 0.71

for wmt22-comet-da .

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ Zh		\Block [c]1-2 Ru $\Leftrightarrow$ En		\Block [c]1-2 En $\Leftrightarrow$ Es		\Block [l]2-1 Avg.
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-12 Llama-3.1-8B-Instruct
Base	19.25	20.03	36.37	27.08	22.89	11.56	40.31	22.21	42.15	40.25	28.21
+ SFT	21.55	39.36	44.20	28.54	30.62	15.43	41.65	23.78	42.60	43.49	33.12
+ CPO	30.98	38.77	44.02	28.27	34.98	16.69	42.11	24.45	43.14	43.52	34.69
+ StarPO	32.88	40.20	44.36	28.77	35.90	17.94	42.45	25.72	43.99	43.68	35.59

Table 3: Performance in d-BLEU score on the News-Commentary test set. Unlike dCOMET, d-BLEU evaluates each document as a single continuous string without sentence alignment. Blue text background denotes a significant improvement over the Base model (

\geq 3.35

BLEU).

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 Zh $\Leftrightarrow$ De		\Block [c]1-2 Zh $\Leftrightarrow$ Ru
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-7 Llama-3.1-8B-Instruct
Base	53.66	64.33	52.31	66.36	60.28	62.04
+ SFT	62.70	65.42	67.96	73.33	75.27	72.75
+ CPO	63.01	68.70	69.04	73.95	75.22	72.92
+ StarPO	63.43	72.17	72.15	74.61	77.50	73.79

Table 4: Performance in dCOMET on Guofeng test set.

Method	Zh $\Rightarrow$ En	En $\Rightarrow$ Zh
\Block [l]1-3 Data Ranking Metrics
BLEU Ranking	76.55	79.71
COMET Ranking	81.01	79.78
COMETKiwi Ranking	80.56	79.73
Word-level Coverage	75.94	79.46
\Block [l]1-3 Online RL Strategies

Table 5: COMET scores comparison against alternative optimization strategies. Word-level Coverage is calculated via WSPAlign Wu et al. (2023) ; Wu et al. (2024c) at the aligned sentence-pair level.

\Block 2-1 Methods	\Block 1-2 Correlation ( $\rho,\uparrow$ )
	Strict $1$ -to- $1$	Relaxed $N$ -to- $M$
\Block [l]1-3 Existing Alignment Methods
Align-then-Slide	0.3804	0.5168
SEGALE	0.4193	0.4760
\Block [l]1-3 Simple Metrics
Token Count Ratio	0.0218	0.0211

Table 6: Spearman correlation (

\rho

) between LLM-annotated alignment quality and various evaluation methods (including STAR variants, Align-then-Slide Guo et al. (2025c) , SEGALE Wang et al. (2025c) , and length-based metrics) under both strict

1

-to-

1

and relaxed

N

-to-

M

matching criteria.

Pattern	Pathological		Complex		1-to-1		Overall
Pattern	%	COMET	%	COMET	%	COMET	COMET
Base	3.25	0.00	4.16	71.87	92.59	75.78	73.15
SFT	3.14	0.00	3.13	74.81	93.73	79.45	76.81
StarPO	2.09	0.00	2.12	81.52	95.79	83.33	81.55

Table 7: Fine-grained analysis on Zh

\Rightarrow

En.

	Fluency ( $\uparrow$ )	Content ( $\downarrow$ )	Cohesion ( $\downarrow$ )
\Block [c]1-4 LLaMA-3.1-Instruct
Base	3.67	2.33	1.95
+ SFT	3.94	1.96	1.70
+ CPO	3.97	1.90	1.58
+ StarPO	3.99	1.40	1.27
\Block [c]1-4 Qwen-2.5-7B-Instruct

Table 8: LLM-as-a-judge evaluation results. Higher is better (

\uparrow

); lower is better (

\downarrow

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Alignment score vs. Token Consumption. We compare our proposed StarPO against various baselines, including sentence-level systems, document-level baselines, and proprietary LLMs (e.g., GPT-4o, Deepseek-R1). Note that the x-axis is plotted on a logarithmic scale (

10^{n}

Figure 3: Case study of structural alignment patterns. Green text indicates acceptable linguistic adaptations (e.g., merging for cohesion or splitting for clarity), while red text highlights genuine translation errors. These two scenarios must be carefully distinguished to avoid misjudging stylistic alignment as semantic loss.

Figure 4: Sentence Translation Alignment Rate (STAR) across varying input context lengths from Sent2Sent to more than 2048 tokens. The results are evaluated on Zh

\Rightarrow

En translation of the News Commentary dataset using LLaMA-3.1-8B-Instruct.

Figure 5: The prompt template used for the LLM-as-a-judge implementation of STAR. The final score is calculated strictly based on the ratio of 1-to-1 matches to total alignment units, serving as a high-precision reference for validating our automated metric.

$\bf\tau$	Zh $\Rightarrow$ En	En $\Rightarrow$ Zh
0.00	81.04	80.01
0.05	81.15	79.96
0.10	81.55	80.11
0.15	81.38	80.08

Appendix

Table 9: Effect of

\tau

on Zh

\Leftrightarrow

En translations.

Dataset	#Document Train/Valid/Test	#Document for StarPO	Average Tokens	Max Tokens
De $\Rightarrow$ En	8.4K/150/150	1,008	1,797	6,540
En $\Rightarrow$ De	8.4K/150/150	734	1,066	4,065
Es $\Rightarrow$ En	9.7K/150/150	3,325	1,643	6,293
En $\Rightarrow$ Es	9.7K/150/150	3,937	1,071	4,146
Ru $\Rightarrow$ En	7.3K/150/150	1,744	1,776	7,951
En $\Rightarrow$ Ru	7.3K/150/150	1,491	1,079	5,557

Appendix

Table 10: Statistics of the News-Commentary dataset.

Dataset	#Document Train/Valid/Test	#Document for StarPO	Average Tokens	Max Tokens
Zh $\Rightarrow$ En	22.0K/25/25	2.0K	1,853	12,961
En $\Rightarrow$ Zh	22.0K/25/25	2.0K	1,624	10,956
Zh $\Rightarrow$ De	6.0K/30/30	2.0K	1,927	7,962
De $\Rightarrow$ Zh	6.0K/30/30	2.0K	2,392	9,909
Zh $\Rightarrow$ Ru	6.0K/30/30	2.0K	1,991	6,134
Ru $\Rightarrow$ Zh	6.0K/30/30	2.0K	2,470	7,514

Appendix

Table 11: Statistics of the Guofeng dataset.

Figure 6: The universal prompt template used for document-level translation. To ensure a fair comparison and eliminate prompt engineering variance, we apply this standardized Doc2Doc instruction across all experiments. ¡tgt_lang¿ represents the target language, and ¡src_doc¿ represents the source document, respectively.

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ En		\Block [c]1-2 De $\Leftrightarrow$ Zh		\Block [c]1-2 Ru $\Leftrightarrow$ En		\Block [c]1-2 En $\Leftrightarrow$ Es		\Block [l]2-1 Avg.
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-12 LLaMA-3.1-8B-Instruct
Base	90.31 96.44	87.21 93.65	92.52 94.43	95.41 95.92	78.96 93.55	82.79 89.37	97.05 98.51	93.92 96.46	90.97 96.98	94.06 95.86	90.32 95.12
+ SFT	93.52 95.00	91.75 95.05	95.77 96.39	95.60 97.03	80.39 94.38	93.58 96.28	96.97 98.58	97.58 97.82	77.13 96.04	95.64 96.27	91.79 96.28
+ CPO	95.00 97.05	91.72 96.62	95.99 96.90	95.52 98.76	79.83 97.93	90.41 97.42	97.30 98.91	97.65 97.88	90.64 91.87	97.50 97.82	93.16 97.12
+StarPO	95.62 98.64	93.36 98.31	96.80 97.08	95.59 98.66	80.80 98.36	97.12 97.54	97.27 98.90	98.01 98.08	90.13 92.45	97.63 98.01	94.23 97.60

Appendix

Table 12: Performance in STAR scores on News-Commentary test set. In each cell, the top value is the Strict score and the bottom value (in gray) is the Relaxed score.

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 Zh $\Leftrightarrow$ De		\Block [c]1-2 Zh $\Leftrightarrow$ Ru
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-7 LLaMA-3.1-Instruct
Base	30.12 38.84	57.60 72.33	16.40 23.30	28.65 35.57	18.37 35.53	31.70 44.35
+ SFT	70.94 80.51	61.46 83.67	42.68 78.67	60.62 69.59	78.63 80.89	50.25 62.60
+ CPO	72.80 80.74	74.15 77.01	73.41 80.06	80.96 85.98	75.81 80.03	87.34 98.58
+ StarPO	74.29 82.37	76.84 83.05	75.97 79.39	84.39 88.32	80.34 85.40	87.94 97.39

Appendix

Table 13: STAR scores on Guofeng test set. In each cell, the top value is the Strict score and the bottom value (in gray) is the Relaxed score.

\Block 2-1 System	Ideal	Structural Deviations
	1-to-1	1-to-0	0-to-1	Other
\Block [c]1-5 LLaMA-3.1-8B-Instruct
Base	92.59	2.08	1.17	4.16
+SFT	93.73	2.72	0.42	3.13
+CPO	95.42	0.33	0.75	3.50
+StarPO	95.79	1.95	0.14	2.12

Appendix

Table 14: Calculating STAR scores in Chinese

\Rightarrow

English language direction on News-Commentary test set by Gemini-2.5-Flash using prompts in Figure 5 .

\Block [l]2-1 System	\Block [c]1-2 Zh $\Leftrightarrow$ En		\Block [c]1-2 Zh $\Leftrightarrow$ De		\Block [c]1-2 Zh $\Leftrightarrow$ Ru
	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$	$\Rightarrow$	$\Leftarrow$
\Block [c]1-7 Llama-3.1-8B-Instruct
Base	8.36	16.56	4.41	6.13	13.57	12.68
+ SFT	9.41	17.45	14.83	15.55	19.78	14.09
+ CPO	11.25	19.19	17.24	16.13	21.67	18.45
+ StarPO	12.08	19.60	18.20	20.49	23.24	19.39

Appendix

Table 15: Performance in BLEU on Guofeng test set.

Method	COMET	COMETKiwi
DPO	36.86	22.61
DPO ( w. SFT loss)	80.69	75.60
SimPO	70.48	70.94
ORPO	80.23	74.94
KTO	80.30	75.04
CPO (Standard)	81.10	75.24

Appendix

Table 16: Comparison of different offline preference optimization algorithms.

Figure 7: Histograms of metric scores for STAR (Original) and STAR (Relaxed). The top plot displays the full distribution including perfect matches (score=100

\%

) The bottom plot zooms in by excluding perfect matches, highlighting that STAR (Original) maintains a dense distribution of high-quality candidates, whereas STAR (Relaxed) has sparse coverage in the near-perfect region.

Figure 8: Frequency distribution of unmasked tokens. To provide a clear view of the token preservation statistics, the distribution is normalized

Figure 9: Analysis of training margins. The left panel compares the margins of Ours, DPO with SFT loss, standard DPO loss, and standard CPO loss. The right panel provides a zoomed-in view focusing on the comparison between standard CPO loss and Ours (indicated as “CPO Loss with Mask” in the legend).

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 259 papers

Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation
Mar 23, 2026Ireh Kim, Tesia Sker, Chanwoo KimMachine Translation Document-Level MT
Department of Artificial Intelligence, Korea University, Seoul, South Korea
Scaling Unsupervised Word Alignment to Documents via Structural Constraints
Aug 21, 2026Michelle Wastl, Jannis Vamvas, Rico Sennrich
Department of Computational Linguistics, University of Zurich
Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking
Sep 11, 2026Xiaotian Wang, Youyuan Lin, Zhan Shen +1Document Chunking LLM Fine-Tuning
The University of Tokyo · Riken · Kyoto University +1

Mar 23, 2026J/K move · Enter open · S save

Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation

Ireh Kim, Tesia Sker, Chanwoo Kim

Department of Artificial Intelligence, Korea University, Seoul, South Korea

Machine Translation Document-Level MT LLM Fine-Tuning

Open paper AskPDF arXiv

In Machine Translation, Large Language Models (LLMs) have generally underperformed compared to conventional encoder-decoder systems and thus see limited adoption. However, LLMs excel at modeling contextual information, making them a natural fit for document-level translation tasks where coherence across sentences is crucial. Despite this potential, document-level MT with LLMs faces two key challenges: (1) the scarcity of large-scale, high-quality document-level parallel data; and (2) the propensity of LLMs to introduce hallucinations and omissions during generation. To address these challenges, we propose a two-stage fine-tuning strategy leveraging LLM-augmented document-level data. First, we augment data by converting summarization data into document-level parallel data using a LLM, and then filter it using multiple metrics, leveraging sacreBLEU, COMET, and LaBSE-based cosine similarity-to improve data quality. Finally, we employ a two-stage fine-tuning strategy: first fine-tuning on the abundant sentence-level MT resources, and then on the filtered document-level corpus.

Abstract

Machine Translation

Explore similar work

Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation

Scaling Unsupervised Word Alignment to Documents via Structural Constraints

Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking