Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck

Hongbin Zhang; Kehai Chen; Xuefen Bai; Youcheng Pan; Yang Xiang; Jinpeng Wang; Min Zhang

cs.CLMar 11, 2026

Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck

Authors: Hongbin Zhang, Kehai Chen, Xuefen Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, Min Zhang

Organizations: Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Keeta AI, Meituan, Beijing, China

Abstract

Large language models (LLMs) have become a standard for multilingual evaluation, yet they exhibit a severe systematic translationese bias. In this paper, translationese bias is characterized as LLMs systematically favoring machine-translated text over human-authored references, particularly in low-resource languages. We attribute this bias to spurious correlations with (i) latent manifold alignment with English and (ii) cross-lingual predictability. To mitigate this bias, we propose DIBJudge, a robust fine-tuning framework that learns a minimally sufficient, judgment-critical representation via variational information compression, while explicitly isolating spurious factors into the dedicated bias branch. Furthermore, we incorporate a cross-covariance penalty that explicitly suppresses statistical dependence between robust and bias representations, thereby encouraging effective disentanglement. Extensive evaluations on multilingual reward modeling benchmarks and a dedicated translationese bias evaluation suite demonstrate that the proposed DIBJudge consistently outperforms strong baselines and substantially mitigates translationese bias.

Figures & tables

Figure 1: Translationese Bias Severity of GPT-4o across languages. Languages are sorted by resource availability from low (top) to high (bottom). The trend line illustrates the inverse relationship between resource availability and translationese bias.

Panel of Figure 2 — Figure 2: Correlation analysis of judge preference with confounding factors. (a) Machine win rate decreases monotonically as CAD increases, indicating that judge preference spuriously tracks latent manifold isomorphism with English. (b) SSR distributions exhibit a clear drift between human-win and machine-win cases, showing that the judge systematically favors higher-likelihood outputs. (c) ROC curves confirm that both CAD and SSR reliably predict judge outcomes, reinforcing the attribution that translationese bias is mediated by latent manifold isomorphism with English and high predicative confidence.

Figure 3: Overview of our DIBJudge , which grounded in Equation 3 . (1) employs a robust encoder $g_{\phi_{r}}$ and a bias encoder $g_{\phi_{b}}$ to separate the input $X$ into robust representations $\mathbf{Z}_{r}$ and bias representations $\mathbf{Z}_{b}$ . (2) introduces a variational bottleneck to minimize the mutual information $I(X;Z_{r})$ . (3) leverages the compressed $\mathbf{Z}_{r}$ through LLM Judge optimized using LoRA ( Hu et al., 2022 ) to generate the final output $Y$ by maximizing $I(Y;Z_{r})$ . (4) ensures feature independence by minimizing the dependence $I(Z_{r};Z_{b})$ between the robust and bias branches. (5) explicitly captures spurious attributes $S$ within $\mathbf{Z}_{b}$ by maximizing $I(S;Z_{b})$ through two proxy tasks: (i) cross-lingual alignment contrastive learning and (ii) predictive confidence estimation via log-probability bin classification.

Proprietary Models
	m-RewardBench	RewardBench	MM-Eval
Model	(Avg. 23 langs)	(English)	(Avg. 18 lang)
GPT-4o ( Hurst et al., 2024 )	$85.75\pm\pm 0.42$	$85.96\pm\pm 0.35$	$71.85\pm\pm 0.81$
Gemini-2.5-Flash ( Comanici et al., 2025 )	$88.06\pm\pm 0.49$	$88.83\pm\pm 0.47$	$77.47\pm\pm 0.76$
General Open Models
Qwen2.5-3B-Instruct ( Qwen et al., 2025 )	$66.97\pm\pm 1.12$	$68.99\pm\pm 1.05$	$57.99\pm\pm 1.20$

Table 1: Performance evaluation on multilingual reward benchmarks. Bold indicates the best performance, and underlined indicates the second-best. Statistical significance compared to the best baseline is denoted by † (

p<0.05

) and ‡ (

p<0.01

Figure 4: Bias severity across resource tiers.

\mathcal{S}_{\text{bias}}

(lower is better) on Belebele , Aya , and XL-Sum . DIBJudge reduces bias across all tiers, with average reductions of

80\%

56\%

, and

75\%

, and the strongest improvements in Low-Resource settings. Error bars show std over 3 runs; Avg

\Delta

is relative to Vanilla SFT .

Figure 5: Bias–utility Pareto Frontier. Trade-off between Bias Severity ( $\downarrow$ ; x-axis) and m-RewardBench accuracy ( $\uparrow$ ; y-axis). Each point corresponds to a bottleneck strength $\beta$ (log-scaled, color-coded). The resulting Pareto frontiers are traced by DIBJudge (solid) and the Vanilla IB baseline (dashed). DIBJudge consistently achieves higher accuracy at comparable bias levels across $\beta$ , yielding a uniformly superior bias–utility trade-off. Markers indicate representative SOTA models, which DIBJudge outperforms in terms of lower bias and higher accuracy.

Panel of Figure 6 — Figure 6: Visualization of Latent Representation Disentanglement. t-SNE projections of embeddings for Human (Blue) vs. Machine (Red) text. (a) $\mathbf{Z}_{b}$ clearly separates domains. (b) $\mathbf{Z}_{r}$ shows a mixed distribution, corroborating domain invariance.

	ID	OOD (Unseen Biases)
Method	Trans. $\mathcal{S}_{\text{bias}}$ $\downarrow$	Length $\rho$ $\downarrow$	Self-Pref. $\mathcal{S}_{\text{bias}}$ $\downarrow$
Vanilla SFT	0.247	0.553	0.314
Vanilla IB	0.168	0.482	0.276
DIBJudge	0.083	0.314	0.219

Table 2: Zero-Shot Generalization to Unseen Biases. We evaluate performance on a held-out subset containing biases not encountered during training (Out-of-Distribution). DIBJudge achieves the lowest bias scores across both in-distribution (Translationese) and unseen heuristics (Length, Self-Preference).

Objectives			Metrics
Compression	Bias	Disentangle	$\mathcal{S}_{\text{bias}}(\downarrow)$	Acc. ( $\uparrow$ )
✓			0.124	85.25
	✓		0.150	86.60
		✓	0.053	85.85
✓	✓		0.091	88.20
✓		✓	0.039	88.55

Table 3: Ablation study of the DIB objectives. We report the bias score (

\mathcal{S}_{\text{bias}}

, lower is better) and Accuracy (higher is better). The combination of all terms achieves the best trade-off.

Configuration	None	CLA	LPBC	CLA + LPBC
$\mathcal{S}_{\text{bias}}(\downarrow)$	0.421	0.312	0.279	0.147
Acc. ( $\uparrow$ )	87.12	87.86	88.43	89.18

Table 4: Ablation study of proxy tasks. CLA indicates the cross-lingual alignment proxy task, and LPBC stands for log-probability bin classification proxy task.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Contents
Appendix A	Translationese Bias Evaluation Suite. Language taxonomy, dataset construction, and qualitative examples.
Appendix B	Theoretical Analysis. Information bottleneck bounds, derivation of compression loss function, identifiability discussion, and disentanglement proofs.
Appendix C	Experimental Details. Benchmarks, evaluation metrics, model configurations, and training protocols.
Appendix D	Comprehensive Multilingual Results. Full results on RewardBench, M-RewardBench, and MM-Eval.
Appendix E	Translationese Bias Results. Detailed quantitative translationese bias evaluation results across language pairs.
Appendix F	Additional Experiments and Ablations. Spurious proxy task analysis, sensitivity analysis, method robustness checks, and information leakage validation of disentangle representation.

Appendix

Table 11

Dataset	High-Resource	Mid-Resource	Low-Resource
Aya	Basque, English, Finnish, Hindi, Japanese, Portuguese, Simp. Chinese, Spanish, Arabic, Vietnamese	Bengali, Cebuano, Filipino, Indonesian, Lithuanian, Malay, Tamil, Thai, Ukrainian, Urdu	Amharic, Irish, Kyrgyz, Nepali, Malagasy, Sinhala, S. Pashto, Telugu, Yoruba, Zulu
Belebele	Arabic, English, Finnish, Hindi, Japanese, Korean, Russian, Turkish, Vietnamese, Simp. Chinese	Bengali, Greek, Hebrew, Georgian, Kazakh, Tamil, Thai, Ukrainian, Urdu, Malay	Amharic, Tibetan, Guarani, Kannada, Khmer, Kyrgyz, Burmese, Punjabi, Pashto, Zulu
XL-Sum	Arabic, Simp. Chinese, English, French, Hindi, Japanese, Korean, Russian, Turkish, Vietnamese	Azerbaijani, Bengali, Indonesian, Tamil, Thai, Ukrainian, Urdu, Uzbek	Amharic, Burmese, Hausa, Kyrgyz, Marathi, Nepali, Pashto, Sinhala, Telugu, Welsh

Appendix

Table 5: Classification of evaluated languages across datasets based on Joshi et al. (2020) taxonomy.

Hyperparameter	Value
Base Model	Qwen3 Family ( Yang et al., 2025 )
Optimizer	Adam
Learning Rate	$1\times 10^{-4}$
LR Scheduler	Cosine
Warmup Ratio	0.1
Max Sequence Length	16,384

Appendix

Table 6: Hyperparameter settings for the training experiments.

Proprietary Models
	Chat	Lang. Hallu.	Linguistics	Reasoning	Safety	Avg.
Model	(Accuracy)	(Accuracy)	(Accuracy)	(Accuracy)	(Accuracy)	(Avg. 18 lang)
GPT-4o	$84.20$	$65.40$	$79.15$	$55.30$	$75.20$	$71.85\pm\pm 0.81$
Gemini-2.5-Flash	$88.50$	$70.10$	$82.45$	$63.80$	$82.50$	$77.47\pm\pm 0.76$
General Open Models
Qwen2.5-3B-Instruct	$66.50$	$52.30$	$58.15$	$40.80$	$72.20$	$57.99\pm\pm 1.20$

Appendix

Table 7: Full detailed results by category of MM-Eval. Bold indicates the best performance, and underlined indicates the second-best.

Proprietary Models
	Chat	Chat Hard	Safety	Reasoning	Average
Model	(Accuracy)	(Accuracy)	(Accuracy)	(Accuracy)	(Avg. 23 lang)
GPT-4o	$90.10$	$75.50$	$88.20$	$89.20$	$85.75\pm\pm 0.42$
Gemini-2.5-Flash	$93.40$	$80.25$	$87.80$	$90.80$	$88.06\pm\pm 0.49$
General Open Models
Qwen2.5-3B-Instruct	$76.50$	$48.20$	$70.10$	$73.10$	$66.97\pm\pm 1.12$

Appendix

Table 10: Full detailed results by category of m-RewardBench. Bold indicates the best performance, and underlined indicates the second-best.

Proprietary Models
	Chat	Chat Hard	Safety	Reasoning	Average
Model	(Accuracy)	(Accuracy)	(Accuracy)	(Accuracy)	(English)
GPT-4o	$90.50$	$75.10$	$88.50$	$89.74$	$85.96\pm\pm 0.35$
Gemini-2.5-Flash	$93.80$	$81.20$	$89.10$	$91.22$	$88.83\pm\pm 0.47$
General Open Models
Qwen2.5-3B-Instruct	$82.50$	$41.50$	$74.50$	$77.46$	$68.99\pm\pm 1.05$

Appendix

Table 13: Full detailed results by category of RewardBench (English). Bold indicates the best performance, and underlined indicates the second-best.

High-Resource
Language	Base	Vanilla SFT	Vanilla IB	DIBJudge
Basque	0.081 $\pm$ 0.008	0.088 $\pm$ 0.043	0.056 $\pm$ 0.011	0.042 $\pm$ 0.009
English	0.045 $\pm$ 0.008	0.058 $\pm$ 0.010	0.041 $\pm$ 0.008	0.031 $\pm$ 0.005
Finnish	0.125 $\pm$ 0.010	0.165 $\pm$ 0.028	0.079 $\pm$ 0.015	0.058 $\pm$ 0.005
Hindi	0.089 $\pm$ 0.008	0.076 $\pm$ 0.010	0.048 $\pm$ 0.007	0.036 $\pm$ 0.012
Japanese	0.049 $\pm$ 0.011	0.048 $\pm$ 0.010	0.034 $\pm$ 0.014	0.026 $\pm$ 0.015

Appendix

Table 14: Bias severity by language on Aya dataset ( Singh et al., 2024 ) under perturbed setting

High-Resource
Language	Base	Vanilla SFT	Vanilla IB	DIBJudge
Arabic	0.100 $\pm$ 0.029	0.095 $\pm$ 0.016	0.058 $\pm$ 0.010	0.014 $\pm$ 0.006
English	0.072 $\pm$ 0.005	0.063 $\pm$ 0.004	0.043 $\pm$ 0.014	0.011 $\pm$ 0.005
Finnish	0.086 $\pm$ 0.015	0.067 $\pm$ 0.017	0.049 $\pm$ 0.023	0.012 $\pm$ 0.010
Hindi	0.091 $\pm$ 0.011	0.089 $\pm$ 0.011	0.052 $\pm$ 0.012	0.013 $\pm$ 0.011
Japanese	0.074 $\pm$ 0.009	0.069 $\pm$ 0.012	0.047 $\pm$ 0.008	0.012 $\pm$ 0.006

Appendix

Table 15: Bias severity by language on Belebele ( Bandarkar et al., 2024 ) under perturbed etting

High-Resource
Language	Base	Vanilla SFT	Vanilla IB	DIBJudge
Arabic	0.079 $\pm$ 0.031	0.100 $\pm$ 0.030	0.084 $\pm$ 0.029	0.025 $\pm$ 0.008
English	0.060 $\pm$ 0.006	0.076 $\pm$ 0.008	0.063 $\pm$ 0.022	0.019 $\pm$ 0.005
French	0.054 $\pm$ 0.010	0.095 $\pm$ 0.019	0.069 $\pm$ 0.009	0.021 $\pm$ 0.005
Hindi	0.101 $\pm$ 0.010	0.126 $\pm$ 0.006	0.083 $\pm$ 0.029	0.024 $\pm$ 0.006
Japanese	0.067 $\pm$ 0.009	0.066 $\pm$ 0.026	0.064 $\pm$ 0.025	0.020 $\pm$ 0.008

Appendix

Table 16: Bias severity by language on XL-Sum ( Hasan et al., 2021 ) under perturbed setting

Translation System	BLEU $\uparrow$	Bias Severity $\downarrow$
NLLB-200-3.3B ( Costa-Jussà et al., 2022 ) (Default)	52.4	0.15
Qwen3-4B ( Yang et al., 2025 )	54.1	0.16
Gemma-3-4B ( Team et al., 2025 )	55.8	0.14
Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 )	56.2	0.18
Gemini-2.5-Flash ( Comanici et al., 2025 )	57.9	0.15
GPT-4o ( Hurst et al., 2024 )	58.5	0.17

Appendix

Table 17: Impact of Back-Translation Systems. We report BLEU scores of the generated back-translations and the resulting Bias Severity.

Metric	Bias Severity $\downarrow$
Type-Token Ratio (TTR)	0.17
Perplexity (PPL)	0.16
Negative Log-Likelihood (NLL)	0.15

Appendix

Table 18: Comparison of Heuristic Metrics. Ablation of the metric used for Bin Classification. NLL performs slightly better, but all metrics provide similar bias mitigation.

Figure 7: Sensitivity of Win Rate to Cross-Lingual Alignment Discrepancy (CAD). Win rates of Translationese outputs are plotted against binned CAD values. Baseline judges (dashed lines) display a strong positive dependence on CAD, reflecting an English-anchoring bias toward structurally aligned translations. In contrast, DIBJudge (solid line) maintains a near-constant win rate across CAD regimes, including high-CAD regions, indicating robustness to cross-lingual structural divergence.

Figure 8: Bias Spectrum Shift via Sequence Surprisal Ratio (SSR). Kernel density estimates of SSR for instances where the judge selects the machine-generated output ( Machine Wins ). Lower SSR values correspond to spurious correlations with high predictive confidence (i.e., low perplexity), indicating a preference for distributionally simple text. Values near

1

reflect invariance to confidence-related artifacts. While the baseline judge concentrates mass at low SSR values, DIBJudge shifts the spectrum toward

1

, demonstrating reduced reliance on predictive-confidence shortcuts.

Method	Constraint Type	m-RB Acc. $\uparrow$	Bias Sev. $\downarrow$
Low-Rank	Deterministic (Capacity)	88.2	0.185
Noise	Stochastic (Additive)	85.5	0.182
VQ	Discrete (Structural)	84.9	0.082
Ours	Variational (Information)	89.3	0.091

Appendix

Table 19: Comparison of Bottleneck Mechanisms. Effect of different latent constraints on utility (m-RewardBench accuracy) and bias mitigation (Bias Severity). Lower Bias Severity is better.

Method	Computational Cost	Latency $\uparrow$	Bias Severity $\downarrow$
Baseline (No disentanglement)	–	$1.00\times$	0.284
$\mathcal{L}_{\text{orth}}$ (Orthogonality)	$O(d)$	1.05 $\times$	0.215
MINE ( Belghazi et al., 2018 )	$O(\text{aux.\ net})$	2.42 $\times$	0.145
CLUB ( Cheng et al., 2020 )	$O(\text{aux.\ net})$	2.12 $\times$	0.141
HSIC ( Gretton et al., 2005 )	$O(n^{2})$	2.78 $\times$	0.138
$\mathcal{L}_{\text{cov}}$ (Ours)	$O(d^{2})$	1.18 $\times$	0.142

Appendix

Table 20: Efficiency–bias trade-off of disentanglement mechanisms on Llama-3-8B. Training latency is normalized to standard fine-tuning (

1.0\times

). Bias Severity is defined in § 2 , where lower is better.

Model	Probed Representation	Accuracy (%)
Baseline (SFT)	Standard embedding $\mathbf{h}$	82.4
DIBJudge (Ours)	Bias representation $\mathbf{z}_{b}$	96.1
DIBJudge (Ours)	Robust representation $\mathbf{z}_{r}$	50.3

Appendix

Table 21: Linear probing for domain classification. Higher accuracy indicates stronger domain information. Effective disentanglement yields high accuracy for bias representations and low accuracy for robust representations.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 118 papers

May 27, 2026

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Large language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for multilingual evaluation, extending LLM-based evaluators to multilingual settings remains challenging, particularly for low-resource languages and scenarios where in-domain data is scarce. This work explores several strategies for developing multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge.

Irune Zubiaga, Aitor Soroa, Rodrigo Agerri

HiTZ Center - Ixa, University of the Basque Country EHU

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Generation Evaluation LLM Fine-Tuning Low-Resource Language Processing

Ask PDF

Apr 21, 2026

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5

York University · University of Alberta · Nanyang Technological University +1

Multimodal Model Evaluation Multilingual VLM Evaluation Multimodal Reward Modeling VLM Evaluation Vision-Language Model-as-a-Judge Cross-Lingual Consistency in Language Models+1

Ask PDF

Jun 6, 2026

When Languages Disagree: Self-Evolving Multilingual LLM Judges

Multilingual LLM-as-a-judge is widely used to evaluate model outputs across languages, but suffers from cross-lingual inconsistency (Fu and Liu, 2025). Existing methods typically treat this inconsistency as noise and mitigate it through voting or aggregation. In this work, we instead show that multilingual inconsistency can provide complementary evaluation signals. Our oracle analysis finds that sampling judgments across languages yields a higher performance upper bound than single-language judging, indicating that different languages potentially include complementary judgments. Motivated by this finding, we propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement. SEMJ constructs multilingual variants of each input, collects independent judgments and rationales, and feeds inconsistent outputs back for self-reflection and re-evaluation. Experiments on multiple benchmarks show that SEMJ consistently outperforms voting and reflection baselines in both accuracy and cross-lingual consistency. Further analysis shows that inconsistency triggers useful re-evaluation, which improves judgment quality.

Xiyan Fu, Wei Lu

Nanyang Technological University

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Self-Improvement Cross-Lingual Consistency in Language Models

Ask PDF

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Lost in Translation: Do LVLM Judges Generalize Across Languages?

When Languages Disagree: Self-Evolving Multilingual LLM Judges