Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

Ivaxi Sheth; Zeno Jonke; Amin Mantrach; Saab Mansour

cs.CLMar 19, 2026

Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

Authors: Ivaxi Sheth, Zeno Jonke, Amin Mantrach, Saab Mansour

Organizations: CISPA Helmholtz Center for Information Security · Amazon

Abstract

As large language models are increasingly deployed across diverse real-world applications, extending automated evaluation beyond English has become a critical challenge. Existing evaluation approaches are predominantly English-focused, and adapting them to other languages is hindered by the scarcity and cost of human-annotated judgments in most languages. We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS). UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation that supports cross-lingual transfer with minimal supervision. Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements over strong baselines without requiring target-language annotations.

Figures & tables

Figure 1: UCS framework. The LLM evaluates an input using criteria questions derived from higher-level concepts. Criterion scores are aggregated into a latent representation, which a transfer module trained on English-labeled data maps to the final judgment and applies across languages.

Method	MEMERAG					mFACE
	DE	FR	ES	HI	Avg	AM	MY	FR	SW	TH	AR	HI	ES	Avg
Zero-shot	77.2	77.5	77.5	74.4	76.7	54.7	70.2	74.3	73.0	72.6	69.7	68.0	77.9	70.1
CoT	73.0	76.2	76.2	74.1	74.9	58.7	68.5	72.8	77.5	71.4	69.2	69.4	74.4	70.2
AG	77.2	80.6	76.1	78.0	78.0	59.3	67.4	76.6	77.7	72.6	70.7	69.0	79.5	71.6
ChatEval	71.8	73.0	60.9	62.4	67.0	51.1	65.5	68.4	74.0	63.9	66.1	55.7	70.3	64.4
CheckEval	76.1	80.7	74.8	78.4	77.5	60.8	68.9	74.9	76.8	75.1	73.4	65.8	76.4	71.5

Table 1: Balanced Accuracy (BA) across methods and languages for the Qwen-235B model. Bold values indicate the best performance per column averaged across 3 runs. Our method UCS is trained on English only data (EN).

Panel of Figure 2 — Figure 2: Sample efficiency of training the transfer module on English data. Performance is shown as a function of the percentage of English training data used. Results are reported for MEMERAG (a) and mFACE (b).

Panel of Figure 3 — Figure 3: Relationship between English–target criteria importance alignment and performance change when training with only the top-10 English-selected criteria.

Method	MEMERAG	mFACE
LogReg	80.9	78.2
SVM	78.5	79.0
KNN	77.1	70.9
RF	76.4	69.8
XGBoost	74.3	71.1
NN	81.5	78.9

Table 2: BA averaged across languages for different transfer modules on the Qwen-235B model.

Method	MEMERAG	mFACE
UCS (Trained)	81.5	74.9
UCS (LLM)	77.2	74.3

Table 3: Average BA across languages on MEMERAG and mFACE. UCS (Trained) uses a lightweight transfer module trained on English-labeled data; UCS (LLM) uses the LLM itself as a training-free aggregator over the criteria responses.

Generation	Scoring	MEMERAG	mFACE
Joint	Joint	79.1	77.4
Joint	Per-concept	81.0	78.6
Per-concept	Joint	80.2	78.0
Per-concept	Per-concept	81.5	78.9

Table 4: Average BA across languages on MEMERAG and mFACE for UCS under different decompositions. Joint denotes evaluating all criteria in one LLM call, while Per-concept denotes one call per concept.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	MEMERAG (BA)				mFACE (BA)
Model	Method	DE	FR	ES	HI	AM	MY	FR	SW	TH	AR	HI	ES
OSS-20B	Zero-shot	70.7	81.6	80.4	81.7	59.4	65.6	73.7	79.2	76.8	69.2	68.5	78.0
	CoT	70.5	81.6	82.0	80.6	60.9	64.9	74.6	76.2	78.3	65.2	67.6	78.0
	AG	71.7	80.8	75.6	80.9	61.4	62.2	75.2	77.8	76.9	72.4	66.7	78.0
	ChatEval	64.2	73.5	71.4	72.0	54.7	60.1	66.9	70.3	68.4	62.9	61.4	71.1
	CheckEval	72.5	81.9	79.9	80.8	60.9	68.3	77.9	76.7	79.7	68.0	65.7	77.0

Appendix

Table 5: Balanced Accuracy (BA) of all baselines and criteria set judges across languages on MEMERAG and mFACE.

Panel of Figure 4 — Figure 4: Relationship between English–target criteria importance alignment.

Method	DE	FR	ES	HI
Zero-shot	77.2 $\pm$ 0.4	77.5 $\pm$ 1.3	77.5 $\pm$ 0.9	74.4 $\pm$ 1.1
CoT	73.0 $\pm$ 1.2	76.2 $\pm$ 0.5	76.2 $\pm$ 1.4	74.1 $\pm$ 0.8
AG	77.2 $\pm$ 0.7	80.6 $\pm$ 1.1	76.1 $\pm$ 0.4	78.0 $\pm$ 1.4
ChatEval	71.8 $\pm$ 1.5	73.0 $\pm$ 0.6	60.9 $\pm$ 1.9	62.4 $\pm$ 0.7
CheckEval	76.1 $\pm$ 0.5	80.7 $\pm$ 1.4	74.8 $\pm$ 0.7	78.4 $\pm$ 1.1
RocketEval	76.4 $\pm$ 1.9	81.0 $\pm$ 1.2	75.9 $\pm$ 1.7	79.1 $\pm$ 1.4

Appendix

Table 6: Balanced Accuracy (BA) on MEMERAG for Qwen-235B, reported as mean

\pm

standard deviation over 3 runs.

Method	AM	MY	FR	SW	TH	AR	HI	ES
Zero-shot	54.7 $\pm$ 0.7	70.2 $\pm$ 1.3	74.3 $\pm$ 0.6	73.0 $\pm$ 1.1	72.6 $\pm$ 0.4	69.7 $\pm$ 1.4	68.0 $\pm$ 1.0	77.9 $\pm$ 0.6
CoT	58.7 $\pm$ 1.4	68.5 $\pm$ 0.6	72.8 $\pm$ 1.2	77.5 $\pm$ 0.5	71.4 $\pm$ 1.1	69.2 $\pm$ 0.5	69.4 $\pm$ 1.6	74.4 $\pm$ 0.8
AG	59.3 $\pm$ 0.8	67.4 $\pm$ 1.5	76.6 $\pm$ 0.7	77.7 $\pm$ 1.1	72.6 $\pm$ 0.6	70.7 $\pm$ 1.2	69.0 $\pm$ 0.7	79.5 $\pm$ 1.2
ChatEval	51.1 $\pm$ 1.6	65.5 $\pm$ 0.9	68.4 $\pm$ 1.3	74.0 $\pm$ 0.5	63.9 $\pm$ 1.8	66.1 $\pm$ 0.6	55.7 $\pm$ 1.1	70.3 $\pm$ 1.5
CheckEval	60.8 $\pm$ 1.3	68.9 $\pm$ 0.4	74.9 $\pm$ 1.6	76.8 $\pm$ 0.7	75.1 $\pm$ 1.1	73.4 $\pm$ 0.8	65.8 $\pm$ 1.5	76.4 $\pm$ 0.6
RocketEval	61.2 $\pm$ 2.1	69.2 $\pm$ 1.1	75.4 $\pm$ 1.7	77.0 $\pm$ 1.3	74.9 $\pm$ 1.6	72.9 $\pm$ 2.0	66.9 $\pm$ 1.4	78.1 $\pm$ 1.8

Appendix

Table 7: Balanced Accuracy (BA) on mFACE for Qwen-235B, reported as mean

\pm

standard deviation over 3 runs.

Dataset	Lang.	UCS (transfer)	UCS (LLM)
MEMERAG	DE	79.7	76.9
	FR	84.3	78.9
	ES	80.0	77.2
	HI	82.2	75.9
mFACE	AM	64.7	62.5
	MY	67.7	69.6

Appendix

Table 8: Comparison of threshold-based training classifier and LLM-based aggregation of intermediate criteria representations for final judgment.

Explore similar work

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
May 27, 2026Irune Zubiaga, Aitor Soroa, Rodrigo AgerriMultilingual Language Model Evaluation LLM-as-a-Judge
HiTZ Center - Ixa, University of the Basque Country EHU
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5VLM Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
When Languages Disagree: Self-Evolving Multilingual LLM Judges
Jun 6, 2026Xiyan Fu, Wei LuMultilingual Language Model Evaluation LLM-as-a-Judge
Nanyang Technological University

May 27, 2026J/K move · Enter open · S save

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Irune Zubiaga, Aitor Soroa, Rodrigo Agerri

HiTZ Center - Ixa, University of the Basque Country EHU

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Generation Evaluation LLM Fine-Tuning Low-Resource Language Processing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for multilingual evaluation, extending LLM-based evaluators to multilingual settings remains challenging, particularly for low-resource languages and scenarios where in-domain data is scarce. This work explores several strategies for developing multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge.

Abstract

Explore similar work

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Lost in Translation: Do LVLM Judges Generalize Across Languages?

When Languages Disagree: Self-Evolving Multilingual LLM Judges