*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Quentin Lemesle; Léane Jourdan; Daisy Munson; Pierre Alain; Jonathan Chevelu; Arnaud Delhay; Damien Lolive

cs.CLFeb 17, 2026

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Authors: Quentin Lemesle, Léane Jourdan, Daisy Munson, Pierre Alain, Jonathan Chevelu, Arnaud Delhay, Damien Lolive

Organizations: Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France · Univ of South Brittany, CNRS, IRISA, ARCHIMEDIA, 56000 Vannes, France

Abstract

Evaluating the quality of automatically generated text often relies on LLM-as-a-judge (LLM-judge) methods. While effective, these approaches are computationally expensive and require post-processing. To address these limitations, we build upon ParaPLUIE, a perplexity-based LLM-judge metric that estimates confidence over ``Yes/No'' answers without generating text. We introduce *-PLUIE, task specific prompting variants of ParaPLUIE and evaluate their alignment with human judgement. Our experiments show that personalised *-PLUIE achieves stronger correlations with human ratings while maintaining low computational cost.

Figures & tables

Task	Paraphrase Classification					Nile Translation
Metric	Thr.	Acc.	Rec.	Prec.	F1	Thr.	Acc.	Rec.	Prec.	F1
*-PLUIE Phi	-7.63	0.71	0.77	0.54	0.63	-3.14	0.81	0.95	0.78	0.85
*-PLUIE Phi	0	0.74	0.54	0.61	0.58	0	0.81	0.85	0.84	0.85
*-PLUIE Llama	-4.07	0.67	0.71	0.50	0.59	3.97	0.80	0.92	0.79	0.85
*-PLUIE Llama	0	0.70	0.64	0.53	0.58	0	0.79	0.94	0.77	0.84
Para-PLUIE Phi	-7.47	0.73	0.73	0.57	0.64	-3.43	0.81	0.87	0.83	0.85

Table 1: Natural and a posteriori optimal threshold for each metric and model-prompt variant, with corresponding accuracy, precision, recall, and F1-score on classification. *-PLUIE is respectively Fr-PLUIE and Net-PLUIE. Confidence interval of reported accuracy, precision, recall, and F1-score is less than

10^{-3}

Task	Nile Translation			Scientific Text Revision
Metric	Pair acc.	$V$	$\kappa$	Pair acc.		$V$		$\kappa$
*-PLUIE Phi	0.69	0.42	0.40	0.61	0.61 w.g.	0.31	0.32 w.g.	0.32	0.33 w.g.
*-PLUIE Llama	0.70	0.43	0.42	0.61	0.62 w.g.	0.31	0.32 w.g.	0.32	0.34 w.g.
Para-PLUIE Phi	0.70	0.44	0.42	0.52		0.21		0.15
Para-PLUIE Llama	0.72	0.46	0.43	0.52		0.20		0.17
LLM-choice Phi	0.47	0.42	0.20	0.53	0.55 w.g.	0.25	0.27 w.g.	0.24	0.27 w.g.

Table 2: Alignment of automatic metrics with human preferences. Pairwise accuracy and

V

are defined on [0:1] and

\kappa

on [-1:1]. "w.g." indicates that the reference revision is provided. Confidence interval of reported pairwise accuracy,

V

, and

\kappa

score is less than

10^{-3}

	Task	Paraphrase Classification		Nile Translation		Scientific Text Revision
model	Approach	GPUs	Runtime	GPUs	Runtime	GPUs	Runtime
Phi	LLM-Yes/No / LLM-choice	MI300 x1	23 min	MI300 x1	11 min	A100 x1	40 min	41 min w.g.
	LLM-Likert			MI300 x1	10 min	A100 x1	51 min	65 min w.g.
	Para-PLUIE	MI300 x1	3.5 min	MI300 x1	1.8 min	A100 x1	08 min
	*-PLUIE	MI300 x1	3.6 min	MI300 x1	1.4 min	A100 x1	09 min	13 min w.g.
Llama	LLM-Yes/No / LLM-choice	MI300 x2	48 min	MI300 x2	22 min	A100 x2	124 min	127 min w.g.

Table 3: GPU usage of all LLM-as-a-judge approaches on all three tasks using Phi and Llama. Confidence interval of reported times is less than

0.1

minute.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Transformation	Sentence
None	Les enfants ont boulotté tous les gâteaux.
	The children gobbled up all the cakes.
Paronym	Les enfants ont boulonné tous les gâteaux
	The children screwed all the cakes.
Synonym	Les enfants ont mangé tous les gâteaux.
	The children ate all the cakes.

Appendix

Table 4: Characteristic sentence of Tytgat et al. (2024) dataset and its different transformations. English translations are provided in italics ; however, as paronym substitution relies on visual and phonetic similarity, it cannot be easily translated. A simple example of an English paronym pair: “The children decided to eat the desert .” vs. “The children decided to eat the dessert .”

Figure 2: Example of data in the ParaReval dataset

model	syst	thr.	f1	prec.	rec.
levenshtein	source	0.17	0.72	0.63	0.84
bleu	source	0.0	0.7	0.54	1.0
meteor	source	0.02	0.74	0.65	0.85
bertscore	source	0.43	0.73	0.66	0.82
modern_bertscore	source	0.65	0.7	0.54	1.0
Net-PLUIE	source	-1.91	0 .78	0.67	0.93

Appendix

Table 5: Metrics evaluation for the Mistral ( Jiang et al., 2023 ) generated translations and the English source intents. Phi-4 14B is the perplexity model used in Net-PLUIE.

Panel of Figure 3 — Figure 3: Score distribution of Modern BertScore (a) and Fr-PLUIE (b). The blue , orange , red and green curves denote respectively the accuracy , recall , precision and F1-score according to the decision threshold. Emphasis is placed on the maximum F1-score obtained by the metric.

Panel of Figure 4 — Figure 4: Accuracy , recall , precision and F1-score distribution over different threshold values for Modern BertScore (a) and Net-PLUIE (b). Emphasis is placed on the maximum F1-score obtained by the metric.

Panel of Figure 5 — Figure 5: Classification with Levenshtein.

Panel of Figure 6 — Figure 6: Classification with METEOR.

Panel of Figure 7 — Figure 7: Classification with BERTScore.

Panel of Figure 8 — Figure 8: Classification with BLEU.

Panel of Figure 9 — Figure 9: Classification with Modern BertScore.

Panel of Figure 10 — Figure 10: Classification with metric Para-PLUIE with Phi-4 14B.

Panel of Figure 11 — Figure 11: Classification with metric Net-PLUIE with Phi-4 14B.

Judge		Pair acc.		$V$		$\kappa$
Judge			w.g.		w.g.		w.g.
Rev-PLUIE	Llama 3 70B	0.61	0.62	0.31	0.32	0.32	0.34
	Phi-4 14B	0.61	0.61	0.31	0.32	0.32	0.33
	Llama 3 8B	0.58	0.59	0.27	0.29	0.27	0.29
	Mistral 7B	0.55	0.59	0.24	0.28	0.22	0.28
Para-PLUIE	Llama 3 70B	0.52		0.20		0.17

Appendix

Table 6: Alignment of LLM-based metrics with human judgements. Pairwise accuracy and Cramér’s V are defined on [0:1] and Cohen’s Kappa on [-1:1]. w.g. (with gold) indicates that the reference revision is provided. For Para-PLUIE, the column without gold corresponds to the Para-PLUIE scores between the original and generated paragraphs. Perplexity models used are Llama-3, Phi-4, Mistral and GPT-4o ( Grattafiori et al., 2024 ; Abdin et al., 2024 ; Jiang et al., 2023 ; OpenAI et al., 2024 ) .

Nile sentence	Translation with Llama ( Grattafiori et al., 2024 )	Translation with Mistral ( Jiang et al., 2023 )
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	The ResNet and NYURoam wireless networks have a 6 GB per day quota	The ResNet and wireless network have a daily cap of 6 GB per day, from midnight to midnight
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	The ResNet cap is currently 6 GB per day	ResNet and UIC wireless have a daily cap of 6 GB for all incoming and outgoing traffic
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	The ResNet cap is currently 6 GB per day	ResNet and wireless users have a daily quota of 6 GB, with no time restrictions
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	The ResNet cap is currently 6 gigabytes per day	The ResNet cap is currently 6 gigabytes per day for each 24 hour period of combined incoming and outgoing traffic
for endpoint(’guests’) block service(’file transfer’)	Guests are not allowed to run file transfer services	Guests are not allowed to transfer files
for endpoint(’guests’) block service(’file transfer’)	Guests are not allowed to use file transfer protocols	Guests are not allowed to transfer files

Appendix

Table 7: A few examples of bad translations that do not respect implication 5 , both translations are aligned to the same intent expression, but ParaPLUIE does not consider the translations to be paraphrases.

Nile sentence	Translation with Llama ( Grattafiori et al., 2024 )	Translation with Mistral ( Jiang et al., 2023 )
for endpoint(’dorms’) block protocol(’nat’)	NAT is not allowed on the ResNet	NAT is blocked on the ResNet
for endpoint(’dorms’), endpoint(’wireless’) block protocol(’ftp’), protocol(’telnet’), traffic(’peer2peer’), protocol(’http’), service(’irc’), protocol(’dns’), service(’email’)	You may not run any services (e.g., FTP services, telnet, P2P file sharing servers, publishing files via generic file services, etc.) via NYU’s ResNet or NYURoam wireless networks	ResNet and wireless networks are blocked from using FTP, Telnet, P2P, HTTP, IRC, DNS, and email services
for endpoint(’dorms’), endpoint(’wireless’) block protocol(’ftp’), service(’listserver’), traffic(’peer2peer’)	The University of Washington does not allow FTP, Listserv, or Peer-to-Peer traffic on the ResNet or UW Wireless	ResNet and UIC wireless do not allow FTP, listserv, or peer-to-peer traffic
for endpoint(’dorms’), endpoint(’wireless’) block protocol(’ftp’), service(’listserver’), traffic(’peer2peer’)	You may not run FTP, listserver, peer-to-peer, or file sharing/downloading program	ResNet and wireless users cannot use FTP, listserv, or peer-to-peer file sharing
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	Dorms and wireless have a quota of 6 GB per day	The quota for dorms and wireless is 6 GB per day, with no time restrictions
for endpoint(’dorms’), endpoint(’wireless’) set quota(’any’, ’6’, ’gb/d’) start hour(’00:00’) end hour(’23:59’)	Students have a quota of 6 GB per day The quota is reset at midnight	ResNet and wireless users have a daily quota of 6 GB, with no time restrictions

Appendix

Table 8: Bad translations that do not respect implication 6 : One translation is aligned to the Nile expression, both translations are paraphrases, but, on lines 1 to 4, ParaPLUIE does not consider Llama’s translation to be aligned to the Nile expression, and on lines 5 to 7, ParaPLUIE does not consider Mistral’s translation to be aligned to the Nile expression.

	French Paraphrase Detection
Metric	Acc.	Rec.	Prec.	F1	GPUs	Runtime
LLM-Yes/No Phi-4 14B (French)	0.74	0.53	0.61	0.57	MI300 x1	27 min
LLM-Yes/No Llama 3 70B (French)	0.71	0.57	0.55	0.56	MI300 x2	57 min

Appendix

Table 9: Results of LLM-as-a-judge approaches with a French prompt.

	French Paraphrase Classification
Metric	Thr.	Acc.	Rec.	Prec.	F1
Para-PLUIE SmolLM2-135M-Instruct	0	0.67	N/A	0.00	0.00
Para-PLUIE SmolLM2-135M-Instruct	-2.31	0.33	0.33	1.00	0.49
Para-PLUIE SmolLM2-360M-Instruct	0	0.67	0.35	0.03	0.06
Para-PLUIE SmolLM2-360M-Instruct	-1.26	0.33	0.33	1.00	0.49
Para-PLUIE SmolLM2-1.7B-Instruct	0	0.61	0.41	0.44	0.43

Appendix

Table 10: Results of Para-PLUIE approaches with for French paraphrase classification with different perplexity models. In the following order, SmolLM2-Instruct ( Allal et al., 2025 ) , internlm2-chat ( Cai et al., 2024 ) , gemma-2 ( Team et al., 2024 ) , Phi-4-mini-instruct ( Microsoft et al., 2025 ) , Mistral-7B-Instruct ( Jiang et al., 2023 ) , Qwen2.5-Instruct ( Qwen et al., 2025 ) , aya-expanse ( Dang et al., 2024 ) , Llama-3.1-Instruct ( Grattafiori et al., 2024 ) , QwQ ( Team, 2025 ) and c4ai-command-r ( Verga et al., 2024 ) .

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
Aug 31, 2026Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Oct 13, 2026Anubhav Jangra, Adrian de WynterLanguage Model Generation Evaluation Generative AI Evaluation
Columbia University, USA · The University of York, UK
Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
Apr 6, 2026Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar +1Language Model Generation Evaluation Multilingual Language Model Evaluation
Qatar Computing Research Institute, HBKU, Qatar · †UPES, India

Aug 31, 2026J/K move · Enter open · S save

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan, Balaraman Ravindran

Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-a-Judge pipelines are increasingly used to evaluate AI-generated text, based on the assumption that judgments arise from reasoning over candidate responses with respect to a rubric. We show that this assumption warrants further scrutiny. Classifiers trained only on rubric text, without access to any evaluated response, achieve nontrivial predictive performance on judge outputs. This suggests that rubric formulations encode recoverable evaluative signals, allowing scores to be partially anticipated independently of model outputs. Finally, counterfactual perturbations reveal that judges often fail to reliably update their decisions when either the candidate response or the rubric criterion is reversed. Our findings raise concerns about the reliability of rubric-based LLM evaluation and highlight the need for further methodological study of automated evaluation via LLMs.

Abstract

Language Model Generation Evaluation

Explore similar work

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation