VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Models

Chonghan Liu; Yimin Du; Qi An; Xin He; Cunqi Zhai; Fei Tan; Weijia Lin; Xiaochun Gong

cs.CLMar 19, 2026

VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Models

Authors: Chonghan Liu, Yimin Du, Qi An, Xin He, Cunqi Zhai, Fei Tan, Weijia Lin, Xiaochun Gong,

Organizations: 1Qiyuan Tech.

Abstract

Large language models frequently exhibit suboptimal performance on low resource languages, primarily due to inefficient subword segmentation and systemic training data imbalances. In this paper, we propose Variable Entropy Policy Optimization (VEPO), which leverages Reinforcement Learning with Verifiable Rewards to incorporate deterministic structural constraints into the policy alignment process. This framework ensures prescribed sequence length, robust format consistency, and rigorous linguistic well formedness, all enforced during training. Central to our approach is a variable entropy mechanism that enables the model to dynamically calibrate the equilibrium between literal fidelity and semantic naturalness by modulating the exploration exploitation manifold. By integrating entropy tempered advantage estimation with asymmetric clipping, VEPO sustains robust exploration while mitigating policy collapse. Empirical evaluations across 90 FLORES-200, COMET-22, chrF directions demonstrate that VEPO yields substantial improvements in both tokenization efficiency and translation quality, bridging the performance gap for underrepresented languages.

Figures & tables

Figure 1: Comparison of translation quality assessment across different models.

Model	Lang.	Length	Format	Mixing	Overall
VEPO-7B (Ours)	97.8	99.2	92.15	92.3	95.3
Qwen2.5-7B-Instruct	92.0	97.0	51.8	62.8	75.9
Gemma3-27B-IT	97.4	91.6	42.1	90.9	80.5
Qwen-MT-Plus	97.6	99.8	82.5	94.8	93.6
Seed-X-PPO-7B	97.6	79.8	79.0	90.3	86.6
DeepSeek-V3	95.4	95.7	67.6	95.0	88.4

Table 1: Constraint verification performance (%). Our model (top) consistently achieves superior alignment across deterministic structural constraints.

Figure 2: Policy entropy dynamics across RL algorithms and KL regimes.

Figure 3: Response length stability across six RL algorithms and three KL regimes (18 independent runs).

Figure 4: Sensitivity analysis of explicit length penalties. External constraints induce reward instability and entropy oscillations, leading to training divergence.

Figure 5: Results of pairwise ranking comparisons from professional human evaluations of translation samples.

Figure 6: Overall translation performance comparison (Avg.) on FLORES-200 across sacreBLEU, COMET, and chrF metrics.

Model	X $\rightarrow$ E	E $\rightarrow$ X	X $\rightarrow$ X	Avg.
Multilingual Chat Models
Gemma3-27B-IT	36.8/.882/62.9	30.7/.875/54.2	22.3/.847/47.5	24.7/.854/49.7
Qwen3-8B	31.1/.862/58.5	23.3/.799/46.4	14.4/.752/36.8	16.9/.767/39.9
Qwen2.5-7B-Inst.	24.8/.498/19.4	17.4/.455/15.6	9.2/.449/14.5	11.6/.454/15.1
Apertus-8B-Inst.	32.5/.870/60.5	25.7/.802/46.6	15.6/.750/36.9	18.3/.767/40.2
Tower-Plus-9B	28.2/.825/54.7	18.3/.671/37.7	9.8/.615/29.3	12.5/.641/32.7

Table 2: Performance on FLORES-200, COMET-22, chrF

Model	BBH	CMMLU	HellaS.	MMLU
Multilingual Chat Models
Gemma3-27B-IT	55.9	55.9	55.9	56.0
Qwen3-8B	63.8	60.8	26.0	51.3
Qwen2.5-7B-Inst.	54.4	64.1	85.2	40.9
Apertus-8B-Inst.	49.2	45.3	64.2	45.2
Tower-Plus-9B	40.4	57.2	73.1	42.1

Table 3: General reasoning benchmarks (%). Red : baseline.

Figure 7: Token level logit distribution comparison at a critical decision point. VEPO reactivates the paraphrastic manifold, allowing the model to explore refined linguistic mappings that are suppressed by mode collapse in the base SFT model.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category 1	Type	Description
Quality	Accuracy	Mistranslation, omission, over-translation, semantic misunderstanding
	Off-target	Untranslated segments, wrong target language
	Fluency	Natural expression, comprehensibility, professionalism
	Terminology	Consistency, accuracy, omission of terms
	Code-mixing	Unreasonable language mixing within sentences
	Repetition	End-of-sentence, segment, instruction repetition

Appendix

Table 4: Error categories for RM robustness

Source	Target	Model BLEU Scores
Source	Target	Gem3	SeedX	QwenMT	Huny	VEPO	Qw2.5	Qw3	Apert	Aya	Emma	LLa3.1	LLaX3	Mistr	Tower	Google	NLLB
Thai	Malay	0.247	0.250	0.212	0.180	0.243	0.132	0.169	0.187	0.092	0.008	0.014	0.173	0.031	0.131	0.234	0.227
Thai	Khmer	0.078	0.006	0.051	0.090	0.091	0.018	0.042	0.005	0.002	0.077	0.012	0.063	0.002	0.003	0.095	0.072
Thai	Lao	0.174	0.012	0.133	0.007	0.199	0.010	0.067	0.026	0.003	0.006	0.005	0.066	0.003	0.005	0.147	0.174
Thai	Burmese	0.102	0.002	0.135	0.121	0.161	0.013	0.048	0.095	0.001	0.067	0.015	0.047	0.003	0.002	0.116	0.066
Thai	Filipino	0.234	0.020	0.179	0.156	0.214	0.066	0.130	0.169	0.054	0.029	0.091	0.149	0.026	0.092	0.148	0.211

Appendix

Table 5: BLEU scores per source–target language pair.

Source	Target	Model COMET Scores
Source	Target	Gem3	SeedX	QwenMT	Huny	VEPO	Qw2.5	Qw3	Apert	Aya	Emma	LLa3.1	LLaX3	Mistr	Tower	Google	NLLB
Thai	Malay	0.8774	0.876	0.877	0.8729	0.8741	0.5088	0.8537	0.8627	0.7755	0.4776	0.7004	0.8471	0.5788	0.7917	0.8481	0.8572
Thai	Khmer	0.7952	0.4917	0.7807	0.8126	0.8245	0.3895	0.6391	0.5416	0.3694	0.7217	0.5474	0.6985	0.4164	0.3355	0.7682	0.7884
Thai	Lao	0.8223	0.51	0.7957	0.6894	0.8409	0.3552	0.6412	0.5743	0.3659	0.5655	0.5892	0.6166	0.3353	0.4796	0.7615	0.8295
Thai	Burmese	0.8373	0.4567	0.8487	0.8498	0.8673	0.4155	0.6412	0.7807	0.3781	0.7714	0.5436	0.6405	0.388	0.3276	0.7535	0.8205
Thai	Filipino	0.8401	0.655	0.8298	0.8244	0.8353	0.4611	0.7514	0.8136	0.5903	0.5316	0.7156	0.7974	0.5538	0.6924	0.749	0.819

Appendix

Table 6: COMET scores per source–target language pair.

Source	Target	Model chrF Scores
Source	Target	Gem3	SeedX	QwenMT	Huny	VEPO	Qw2.5	Qw3	Apert	Aya	Emma	LLa3.1	LLaX3	Mistr	Tower	Google	NLLB
Thai	Malay	58.31	57.57	55.44	55.74	57.95	23.09	51.79	53.82	42.32	6.43	4.89	50.21	35.48	48.86	58.05	57.71
Thai	Khmer	36.93	2.85	34.34	39.40	42.28	13.25	28.35	15.82	11.60	35.03	18.45	30.47	1.76	11.74	41.98	41.85
Thai	Lao	45.85	4.08	41.23	2.48	48.60	11.28	31.41	18.32	10.51	2.03	4.79	27.54	3.21	9.20	46.82	48.23
Thai	Burmese	39.40	0.71	42.25	44.15	46.55	17.47	29.83	36.38	10.15	31.80	16.20	25.48	15.08	11.84	44.23	44.18
Thai	Filipino	54.12	18.07	49.96	51.67	53.14	23.36	44.68	49.49	36.54	14.57	31.06	46.40	32.88	41.20	48.17	53.69

Appendix

Table 7: chrF scores per source–target language pair.

Role	Utterance
System	You are a helpful assistant.

User	Please translate the following sentence into English: Cho ngap phai ruoi

AssistantA	By sheer chance.

Appendix

Table 8: Example conversation showing four different translation approaches: idiomatic, literal, explanatory, and noisy outputs in response to the same Vietnamese idiom.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation
Mar 13, 2026Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa +1Low-Resource MT Multilingual Language Models
Carnegie Mellon University
Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
May 14, 2026Zeli Su, Ziyin Zhang, Zhou Liu +7Reinforcement Learning Low-Resource MT
Minzu University of China · Ant Group · Shanghai Jiao Tong University +4
LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
Mar 23, 2026Hailay Teklehaymanot, Dren Fazlija, Wolfgang NejdlLow-Resource Language Processing Word Embeddings
L3S Research Center, Hannover, Germany

Mar 13, 2026J/K move · Enter open · S save

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa, Lei Li

Carnegie Mellon University

Low-Resource MT Multilingual Language Models RL for Language Models Reward Hacking

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.

Abstract

RL for Language Models

Explore similar work

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation