Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

Firoj Alam; Gagan Bhatia; Sahinur Rahman Laskar; Shammur Absar Chowdhury

cs.CLApr 6, 2026

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

Authors: Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar, Shammur Absar Chowdhury

Organizations: Qatar Computing Research Institute, HBKU, Qatar · †UPES, India

Abstract

While Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to prompt design, language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose OmniScore, a family of complementary, deterministic learned metrics developed using small size ( $<$ 1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision ( $\sim$ 564k instances, in 107 languages) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in 6 languages. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and datasets can be found at https://huggingface.co/collections/QCRI/omniscore

Figures & tables

Figure 1: OmniScore system architecture and training paradigm. Efficient encoder backbones ( $<1$ B parameters) are trained via distillation from a frontier LLM and evaluate text across four quality dimensions and tasks.

Source	Task	Lang	#items	Source	Task	Lang	#items
ParaSCI	Paraphrase	en	500	Vārta	Headline	asm	500
Flores	MT	en_bn	500	Vārta	Headline	bn	500
Flores	MT	en_hi	500	Vārta	Headline	en	500
NQA	QA	en	500	Vārta	Headline	hi	500
MultiNativQA	QA	bn	501	XLSum	Summ.	ar	530
MultiNativQA	QA	asm	543	XLSum	Summ.	bn	501

Table 1: Statistics of the OmniScore-Bench, human-annotated test set ( 8,617 items), grouped by source dataset, task, and language. MT = machine translation; QA = question answering; Summ. = summarization; ar_qa = Qatar-specific Arabic QA.

Average scores								Agreement (rwg)
Source	Task	n	Inf.	Cla.	Pla.	Fai.	Overall	Source	Task	n	Inf.	Cla.	Pla.	Fai.	Overall
FLORES+	MT	1,000	4.36	4.32	4.46	4.41	4.39	FLORES+	MT	1,000	0.88	0.82	0.89	0.88	0.87
MultiNativQA	QA	2,580	4.47	4.41	4.49	4.42	4.45	MultiNativQA	QA	2,580	0.85	0.86	0.89	0.86	0.86
NQ	QA	500	4.39	4.04	4.07	4.07	4.14	NQ	QA	500	0.88	0.76	0.86	0.85	0.84
ParaSCI	Paraphrase	500	3.95	3.91	3.89	3.6	3.84	ParaSCI	Paraphrase	500	0.71	0.7	0.76	0.77	0.73
Varta	Headline	2,000	4.23	4.47	4.53	4.43	4.41	Varta	Headline	2,000	0.88	0.88	0.88	0.89	0.89

Table 2: Test-set human evaluation statistics grouped by source and task. The top part reports mean annotation scores, and the bottom part reports mean inter-annotator agreement (

rwg

). Inf.=Informativeness, Cla.=Clarity, Pla.=Plausibility, Fai.=Faithfulness. The Overall column is the mean across the four dimensions.

Source	Task	Language	Train	Dev	Total
FLORES+	MT	21 lang.	221,295	0	221,295
MultiNativQA	QA	ar, as, bn, en, hi, tr	44,449	6,119	50,568
NQ	QA	en	49,907	0	49,907
ParaSCI	Paraphrase	en	14,995	0	14,995
Vārta	Headline	15 lang.	32,892	2,832	35,724
WildChat	Chat	76 lang.	76,993	4,992	81,985

Table 3: Summary of the training and development data used in our experiments, covering 107 languages.

Split	Size	QA	Chat	Summ.	Head.	Para.	MT
Train	539,015	17.5	14.3	18.3	6.1	2.8	41.1
Dev	24,524	25.0	20.4	43.1	11.5	0.0	0.0
Test	8,617	35.7	0.0	23.6	23.2	5.8	11.6

Table 4: Task composition by split. Percentages are computed within each split. Total dataset size: 572,156 .

Split	Task	n	Info.	Clarity	Plaus.	Faith.	Avg.	Split	Task	n	Info.	Clarity	Plaus.	Faith.	Avg.
Test	Headline	2,000	4.23	4.47	4.53	4.43	4.41	Train	Chat	76,993	4.05	4.9	4.81	4.56	4.58
Test	Paraphrase	500	3.95	3.91	3.89	3.6	3.84	Train	Headline	32,892	3.76	4.84	4.9	4.45	4.49
Test	QA	3,080	4.46	4.35	4.42	4.36	4.4	Train	Paraphrase	14,995	3.3	4.51	4.72	2.7	3.88
Test	Summ.	2,037	4.24	4.15	4.41	4.21	4.25	Train	QA	94,356	3.29	3.64	4.03	3.56	3.63
Test	Trans.	1,000	4.36	4.32	4.46	4.41	4.39	Train	Summ.	98,484	1.62	4.31	4.58	3.15	3.41
Test	Total	8,617	4.31	4.3	4.42	4.3	4.33	Train	Trans.	221,295	4.56	4.63	4.68	4.11	4.5

Table 5: Average annotation scores by split and task. The Avg. column is the average of the four metrics.

Metric	MXL	DBV3	KLM	MDL	MBT	E32	MBR	Q3E	MMS	E150	G300	E68	ARC	E17	MMB	G3F	MAT	G3L
MAE	0.78	0.79	0.79	0.79	0.80	0.80	0.82	0.82	0.82	0.83	0.83	0.83	0.84	0.84	0.87	0.92	0.93	1.01
RMSE	0.98	0.99	0.99	0.99	1.00	1.01	1.03	1.03	1.03	1.03	1.04	1.02	1.04	1.03	1.07	1.10	1.13	1.25
Acc@ $\,\pm 1$	0.74	0.74	0.73	0.74	0.74	0.73	0.72	0.72	0.73	0.72	0.72	0.70	0.72	0.70	0.69	0.73	0.68	0.70

Table 6: Overall results across models. Lower MAE and RMSE are better, while higher Acc@

\,\pm 1

is better. Short model names are used for readability. MXL = mxbai-large, DBV3 = DeBERTa-v3, KLM = KaLM-mini, MDL = MDBR-leaf, MBT = mBERT, E32 = Ettin-32M, MBR = ModernBERT, Q3E = Qwen3-Emb-0.6B, MMS = mmBERT-small, E150 = Ettin-150M, G300 = Gemma-300M-Emb, E68 = Ettin-68M, ARC = Arctic-m-v2, E17 = Ettin-17M, MMB = mmBERT-base, G3F = Gemini-3-Flash, MAT = Matryoshka-mmBERT, and G3L = Gemini-3.1-Flash-Lite.

Model	Head.	Para.	QA	Summ.	Trans.	Avg.
mxbai-large	0.61	0.86	0.66	1.09	0.68	0.78
DeBERTa-v3	0.64	0.90	0.64	1.10	0.76	0.81
Ettin-150M	0.60	0.92	0.74	1.13	0.79	0.84
mBERT	0.67	0.97	0.64	1.14	0.81	0.85
Gemini-3.1-Flash-Lite	0.89	1.09	1.15	0.91	0.90	0.99

Table 7: Average results across tasks.

Model	ar	as	bn	en	hi	np	Avg.
mxbai-large	0.90	0.80	0.86	0.78	0.66	0.73	0.78
KaLM-mini	0.90	0.80	0.82	0.83	0.71	0.62	0.79
Gemma-300M-Emb	0.93	0.88	0.90	0.83	0.76	0.60	0.83
Gemini-3-Flash	0.76	1.02	1.01	0.99	0.82	0.91	0.92

Table 8: Language-wise MAE by model. Lower is better. Bold marks the best model in each column.

Model	Params	Ctx	MAE $\downarrow$	L-MAE $\downarrow$	Acc@ $\pm 1$ $\uparrow$	Time $\downarrow$	Cost $\downarrow$
mxbai-large	335M	1K	0.78	0.79	0.74	3.24	0.03
DeBERTa-v3	86M	512	0.79	0.79	0.74	4.45	0.01
Ettin-17M	17M	8K	0.84	0.84	0.70	0.13	0.01
Ettin-32M	32M	8K	0.80	0.80	0.73	0.33	0.01
Gemini(L)	–	65k	1.01	1.00	0.70	312	0.35

Table 9: MAE is computed from the overall results. L-MAE: Lang-MAE is the test-size-weighted language-wise MAE. Ctx denotes context size. Time is measured in seconds per 1,000 examples. Cost is reported in US dollars per 1,000 examples and is calculated using H100 pricing from Lambda Labs. Gemini (L): Gemini-3.1-Flash-Lite.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Fam.	Rep.	Det.	Ref	ML	Task
Judge	Bavaresco et al. (2025)	✗	–	Var.	Many
	Tan et al. (2025)	✗	–	–	J-Eval
	Xu et al. (2025a)	✗	–	–	J-Eval
	Sheng et al. (2025)	✗	–	–	J-Eval
Distill.	Zhu et al. (2025)	✓ †	Opt.	–	Gen.
Metric	Zhang et al. (2019)	✓ †	Yes	Part.	Gen.

Appendix

Table 10: Comparison of evaluation approaches relevant to deterministic learned metrics that complement LLM-as-a-Judge. Column abbreviations: Fam. = Family, Rep. = Representative, Det. = Deterministic, Ref = reference-based, ML = multilingual, Var. = varies, Opt. = optional, Part. = partial, Gen. = general, Mix. = mixed, Res. = resources, Sum. = summarization, and J-Eval = judge evaluation.

✓^{\dagger}

indicates determinism conditional on fixed model weights and deterministic decoding, when applicable.

Figure 2: Distribution of human ratings on the test set. For each metric, the figure shows the proportion of test examples assigned scores 1–5 for each task.

Figure 3: Test-set quality and annotation agreement. Each point represents a task–metric pair in the test set. The x-axis shows the average Likert score, and the y-axis shows average inter-annotator agreement ( $rwg$ ).

Figure 4: Task composition in different splits. Training is dominated by translation, while the test set emphasizes QA, summarization, and headline generation.

Model	Architecture	Params	Ctx	Tokenizer	Train tokens
mxbai-large	Embedding encoder	335M	1024	–	–
mdbr-leaf	Distilled encoder	23M	–	–	–
DeBERTa-v3	Encoder-only Transformer	86M †	512	128K vocab	160GB text
KaLM-mini	Qwen2-based embedding model	$\sim$ 0.5B	–	–	–
mBERT-cased	BERT-base encoder	$\sim$ 177M	512	WordPiece, 110K vocab	–
Ettin-17M	Encoder-only Transformer	17M	8K	ModernBERT tokenizer, 50,368 vocab	up to 2T

Appendix

Table 11: Architectural summary of the models used in our experiments. Ctx denotes maximum input context length in tokens.

\dagger

For DeBERTa-v3 , 86M refers to backbone parameters; the embedding layer adds additional parameters due to the 128K vocabulary. mmBERT-test uses the same backbone as mmBERT-base but is restricted to the test-language setting in our experiments.

Model	ar	as	bn	en	hi	np	Avg.
mxbai-large	0.90	0.80	0.86	0.78	0.66	0.73	0.78
DeBERTa-v3	0.87	0.80	0.84	0.82	0.68	0.70	0.79
KaLM-mini	0.90	0.80	0.82	0.83	0.71	0.62	0.79
MDBR-leaf	0.85	0.80	0.86	0.82	0.69	0.75	0.79
mBERT	0.83	0.80	0.84	0.87	0.70	0.67	0.80
Ettin-32M	0.86	0.83	0.87	0.84	0.69	0.66	0.80

Appendix

Table 12: Language-wise MAE by model. Lower is better. Bold marks the best model in each column.

Model	Head.	Para.	QA	Summ.	Trans.	Avg.
mxbai-large	0.61	0.86	0.66	1.09	0.68	0.78
DeBERTa-v3	0.64	0.90	0.64	1.10	0.76	0.81
MDBR-leaf	0.63	0.94	0.66	1.09	0.78	0.82
Ettin-32M	0.65	0.91	0.64	1.15	0.76	0.82
KaLM-mini	0.64	0.91	0.69	1.10	0.81	0.83
Qwen3-Emb-0.6B	0.65	0.90	0.70	1.12	0.79	0.83

Appendix

Table 13: Average results across tasks.

Lang	$n$	Mix	MAE
hi	4005	QA/Hd/Su/Tr	0.66
en	3007	Hd/QA/Su	0.78
as	2090	QA/Hd	0.80
bn	3003	QA/Hd/Su	0.86
ar	3068	QA/Su	0.90
np	1002	QA	0.73

Appendix

Table 14: Language-wise MAE of mxbai-large on the test set. Hd=Headline, QA=Question Answering, Su=Summarization, Tr=Translation. Lower is better.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
Aug 31, 2026Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Oct 13, 2026Anubhav Jangra, Adrian de WynterLanguage Model Generation Evaluation Generative AI Evaluation
Columbia University, USA · The University of York, UK
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Apr 10, 2026Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe +2Language Model Generation Evaluation Generative AI Evaluation
Artefact Research Center · MICS, Centrale-Supélec, Université Paris-Saclay · Diabolocom +1

Aug 31, 2026J/K move · Enter open · S save

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan, Balaraman Ravindran

Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-a-Judge pipelines are increasingly used to evaluate AI-generated text, based on the assumption that judgments arise from reasoning over candidate responses with respect to a rubric. We show that this assumption warrants further scrutiny. Classifiers trained only on rubric text, without access to any evaluated response, achieve nontrivial predictive performance on judge outputs. This suggests that rubric formulations encode recoverable evaluative signals, allowing scores to be partially anticipated independently of model outputs. Finally, counterfactual perturbations reveal that judges often fail to reliably update their decisions when either the candidate response or the rubric criterion is reversed. Our findings raise concerns about the reliability of rubric-based LLM evaluation and highlight the need for further methodological study of automated evaluation via LLMs.

Abstract

Language Model Generation Evaluation

Explore similar work

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation