The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.
Figures & tables
Figure 1: Example of model responses to multilingual prompt after alignment tuning.
Model
Method
Chinese
Indonesian
Korean
Swahili
Bengali
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
Llama-2-7B
SFT+DPO
57.80
58.50
57.92
58.88
54.28
55.40
38.63
38.07
34.90
35.68
CLO
56.41
56.08
53.84
53.91
51.15
50.24
44.49
43.97
33.15
30.93
CRPO
64.18
64.34
63.83
64.47
63.02
63.97
44.07
43.72
37.52
36.77
Llama-3-8B
SFT+DPO
60.24
60.37
59.24
60.37
64.97
65.96
46.23
46.95
34.77
36.89
CLO
58.70
58.13
52.42
52.96
53.94
53.41
43.78
44.93
35.67
33.29
Table 1: AlpacaEval across five target languages. LC and WR denote length-controlled and raw win rate, respectively. The best results for each model are highlighted in bold .
Model
Lang
SFT+DPO
CLO
CRPO
MMMLU
Llama-2-7B
id
26.33
24.69
27.85
ko
26.20
28.57
31.07
sw
25.08
25.62
25.12
Llama-3-8B
id
41.69
36.41
45.94
ko
39.58
40.19
39.84
Table 2: MMMLU and Belebele evaluation results. We report the performance of zero-shot in MMMLU and one-shot in Belebele.
Figure 2: (a) Reward difference distribution and (b) Log-likelihood distribution on chosen responses ( yw ) of Llama-3-8B trained with SFT (blue), CLO (orange), SFT+DPO (blue) and CRPO (red) across languages.
Model
Method
zh
id
ko
sw
Llama-2-7B
SFT
-1.749
-1.797
-2.870
-2.901
SFT+DPO
-1.227
-0.904
-2.121
-2.958
CLO
-1.539
-1.200
-2.855
-2.457
CRPO
-0.607
-0.936
-1.601
-2.175
Llama-3-8B
SFT
0.344
0.208
-0.904
-0.487
SFT+DPO
1.506
1.585
0.869
-0.212
Table 3: Average of reward scores from Skywork-Reward-V2-Qwen3-8B on generation outputs. The best results achieved across the methods are highlighted in bold, while the second-best results are underlined.
Figure 3: Win rate comparison of different weighting schemes within the CRPO framework.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Gain values ( G )
Chinese
Korean
Indonesian
Swahili
Bengali
WC (%)
LC (%)
WC (%)
LC (%)
WC (%)
LC (%)
WC (%)
LC (%)
WC (%)
LC (%)
7, 3, 1, 0
60.50
60.75
66.46
66.04
67.20
66.86
66.46
66.35
54.16
52.54
9, 5, 7, 4
60.00
60.39
66.21
66.11
65.22
65.52
67.14
67.43
53.98
53.79
9, 7, 5, 4
62.36
62.46
68.82
68.46
68.41
67.98
62.17
61.85
55.65
53.87
Appendix
Table 4: WR and LC according to different gain values.
Model
Method
Chinese
Indonesian
Korean
Swahili
Bengali
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
Llama-2-7B
SFT+DPO
55.06
55.71
55.89
56.52
57.79
58.26
56.04
56.46
51.46
52.42
CLO
57.46
57.27
53.49
52.24
55.30
54.91
53.41
51.93
53.22
52.61
CRPO
62.33
62.36
60.41
60.62
60.46
60.87
60.43
60.00
61.60
61.99
Llama-3-8B
SFT+DPO
57.71
58.45
60.31
61.30
56.23
56.83
58.06
58.01
56.98
57.76
CLO
52.83
52.48
56.89
56.21
52.66
50.99
54.46
52.61
56.27
54.78
Appendix
Table 5: English performance in AlpacaEval benchmark. The best results for each model are highlighted in bold .
Figure 4: Reward distribution of chosen response (a) and rejected response (b) of Llama-3-8B trained with SFT (blue), CLO (orange), SFT+DPO (blue) and CRPO (red) across languages.
Model
Method
Chinese
Indonesian
Korean
Llama-3-8B
SFT+DPO
61.56
61.34
64.30
CLO
54.49
54.53
54.95
CRPO
62.15
61.87
68.81
Mistral-7B
SFT+DPO
54.62
54.34
55.39
CLO
55.49
53.34
53.02
CRPO
58.22
63.85
62.63
Appendix
Table 6: Win rate in Arena-Hard benchmark.
Lang
Model
Method
Target
English
WR
LC
WR
LC
zh
Llama-3-8B
PRO
24.97
24.24
37.27
36.85
CRPO
62.36
62.45
57.20
56.73
Mistral-7B
PRO
13.42
13.75
36.15
33.67
CRPO
63.10
62.83
57.08
50.60
id
Llama-3-8B
PRO
23.11
24.79
44.47
43.60
Appendix
Table 7: AlpacaEval results of PRO and CRPO under the same training budget.
Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to 28% win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.
Jiaying Lin, Seongho Son, Nam Phuong Tran +3
Department of Computer Science, University of Warwick, Coventry, United Kingdom · Department of Computer Science, University College London, London, United Kingdom · Department of Mathematics and Computer Science, University of Basel, Basel, Switzerland
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, evaluated by two judges, both tuned models win 28/30 times against their respective base across 15 evaluated languages (high and low-resource). Overall, we show promising directions for multilingual preference tuning using self-generations.
Mike Zhang, Ali Basirat, Desmond Elliott
Department of Computer Science (DIKU), University of Copenhagen · Centre for Language Technology (CST), University of Copenhagen · Pioneer Centre for Artificial Intelligence
Alignment of large language models typically relies on pairwise preference feedback, though ranked feedback can capture more comprehensive information across multiple responses within a single annotation. However, ranked-preference alignment is underexplored in practice compared to simpler pairwise approaches, potentially resulting in suboptimal behavior. Concurrently, recent research has conceptualized large language model alignment objectives as divergence estimators between aligned (preferred) and unaligned (less-preferred) response distributions, resulting in a family of objectives with an interpretable theoretical foundation and strong empirical performance. This work unifies these perspectives by introducing f-RPO, a f-divergence objective for ranked preference optimization that extends divergence-based alignment to ranked feedback and recovers pairwise preference alignment as a special case. Our theoretical analysis establishes rank-order consistency, showing that f-RPO preserves the underlying preference ordering at convergence. Empirical results indicate that total-variation and reverse-KL variants of the proposed objective consistently outperform ranked-alignment baselines in both ranking and generation quality across Qwen, Llama, & Gemma models ranging from 0.5B to 8B parameters.