Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Anubhav Jangra; Adrian de Wynter

cs.CLOct 13, 2026

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Authors: Anubhav Jangra, Adrian de Wynter

Organizations: Columbia University, USA · The University of York, UK

Abstract

Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.

Figures & tables

Language	Train	Dev	Test	mean (sd)	$\alpha$	mean (sd)	$\alpha$	mean (sd)	$\alpha$	mean (sd)	$\alpha$
	Split			Style $\uparrow$		Factual $\downarrow$		Leakage $\downarrow$		Usability $\downarrow$
Bengali (bn)	801	101	100	1.24 (1.25)	0.95	0.06 (0.36)	0.78	0.45 (0.83)	0.98	2.02 (1.33)	0.82
English (en)	798	101	103	1.52 (0.82)	0.78	0.23 (0.71)	0.98	0.19 (0.53)	0.99	2.75 (0.99)	0.84
Filipino (fil)	802	101	99	1.16 (1.37)	0.58	1.51 (1.88)	0.85	0.50 (0.86)	0.80	2.81 (1.42)	0.58
French (fr)	802	100	100	1.17 (1.10)	0.95	0.16 (0.59)	0.68	0.54 (0.88)	0.89	2.81 (1.15)	0.87
Frisian (fy)	814	102	102	2.28 (1.35)	0.63	0.40 (1.13)	0.68	0.38 (0.78)	0.91	2.32 (1.31)	0.71

Table 1: Dataset and annotation statistics for the curated multi-lingual style personalized text generation benchmark. For each evaluation dimension, we provide the aggregate human label mean (standard deviation), and the ordinal Krippendorff

\alpha

. The arrows indicate the direction for improved generation quality in each dimension. The final row provides an aggregate across all ten languages.

	Pointwise (Spearman $\rho$ )					Pairwise (Cohen’s $\kappa$ )
Metric ( $k$ )	Style	Factual	Leakage	Usability	Overall	Style	Factual	Leakage	Usability	Overall
Levenshtein	0.06	0.41	-0.51	–	0.33	0.13	0.75	0.81	–	0.56
Verbatim	0.03	-0.38	0.55	–	0.32	-0.01	0.69	0.87	–	0.52
LaBSE	0.07	0.41	-0.57	–	0.35	0.11	0.76	0.85	–	0.57
mStyleRepr (3)	0.01	0.50	-0.58	–	0.36	-0.11	0.83	0.80	–	0.50
Gemma4-12B	0.39	0.50	0.72	0.48	0.52	0.49	0.91	0.96	0.60	0.74

Table 2: Pairwise and pointwise statistical diagnostics over the four evaluation dimensions and their overall aggregate. The evaluation is performed on the test split of the dataset, averaged over human annotations from 10 languages. The pointwise alignment is computed with Spearman’s

\rho

, measured against the mean ordinal human ratings for each paraphrase on the respective evaluation dimension. Whereas the pairwise alignment is measured with Cohen’s

\kappa

, against the sign of the difference in ordinal ratings for the two paraphrases, dropping the instances where the difference is zero. This table includes only results for the best-performing metrics in our meta-evaluation (Figure 2 ); for pointwise and pairwise results across all metrics, refer to Table 6 .

Figure 1: The aggregate

\Delta R^{2}_{p|.}

(Eq. 1 ) for selective evaluation metrics over the four evaluation dimensions against the rest of the evaluation metrics (excluding style embeddings due to their limited language). For the results on all the metrics, refer to Figure 6 in Appendix C .

Figure 2: Meta-evaluation of metrics evaluated using the normalized quality score

\Psi

(Eq. 2 ), averaged over 10 languages.

\Psi^{e}=0

indicates the chance performance, whereas

\Psi=1

corresponds to the average human performance. We report the results for the best-performing individual metrics in the corresponding evaluation paradigm as well as several combinations of ensemble metrics (

\rho

). We provide the detailed results for all the metrics in Tables 7 and 8 in Appendix C .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Performance of 16 regression models spanning four regression approaches. The regression models were trained to fit the metric outputs from all 32 evaluation metrics ( i.e., the

\rho_{all}

ensemble) to predict the reliability-weighted human annotations in a 5-fold cross-validation repeated five times on 10 languages for four evaluation dimensions ( style accuracy, factual accuracy, leaks and usability ).

Composition of the English corpus	Count	%
Reference (style source) domain
r/ShortStories	857	85.5
Amazon Reviews (Digital Music)	121	12.1
Amazon Reviews (Subscription Boxes)	24	2.4
Origin (content source) domain
r/ShortStories	341	34.0

Appendix

Table 3: Composition of the English corpus from which the multilingual SPTG corpus is built (

N{=}1{,}002

annotated instances, drawn from the

1{,}502

translated source instances). Each instance pairs a reference (a set of style exemplars) with an independently sampled origin (the content to be re-expressed), and the two are matched at random. References are drawn mostly from creative short-form writing (r/ShortStories; https://www.kaggle.com/datasets/trevordu/reddit-short-stories ), with the remainder from Amazon product reviews ( Hou et al., 2026 ) ; origins span fiction, encyclopedic text (Wikipedia), movie reviews (IMDB; Maas et al., 2011 ), product reviews, and news (CNN/DailyMail; Hermann et al., 2015 ).

Language	Items	Edited (%)	Post-edit % ( $\mu\pm\sigma$ )	Notes
English (en-US)	1,002	–	–	–
French (fr-FR)	1,002	98	36.2 $\pm$ 9.7	0
Zulu (zu)	1,002	76	32.8 $\pm$ 24.3	0
German (de)	1,002	92	27.0 $\pm$ 14.9	1
Maltese (mt)	1,002	94	26.5 $\pm$ 20.5	1,805
Danish (da)	1,002	96	17.0 $\pm$ 12.5	5

Appendix

Table 4: Post-editing statistics for LLM-generated translations. The native speakers were given the original English and model-translated outputs to make corrections. For each locale, we report the percentage of translations that the post-editors changed ( Edited ), the extent of these changes as the character-level edit distance between the machine translation and its corrected version, normalized by length ( Post-edit ; mean

\pm

s.d. in %), and the total number of free-text comments left by the annotators ( Notes ). Locales are sorted by post-edit distance.

Correction type	Notes (%)	Representative annotator notes
Fluency / phrasing	576 (11)	[Irish] rephrased to i gcuid mhaith , which “sounds more natural”; [Maltese] removed a duplicated li in the second sentence; [Hebrew] first sentence “feels odd both in English and in Hebrew”.
Lexical / word choice	400 (7)	[Irish] “NOLA” expanded to “New Orleans”; [Welsh] (g)wraig (“wife”) replaced by menyw (“woman”).
Grammar / morphology	232 (4)	[Welsh] agosawu corrected to agos (“close”); [Maltese] verb aspect ninhass $\rightarrow$ qed inxomm , past tense insista $\rightarrow$ insistejt .
Spelling / typo	218 (4)	[Welsh] Stifft corrected to stiff ; [Welsh] corrected the spelling of “Volkswagen”.
Mistranslation / wrong sense	121 (2)	[Welsh] “crackling” rendered as cyfarth (“barking”), corrected to cracio ; [Welsh] dioddefaint (“suffering”) used where poen (“pain”) was meant; [Hindi] “Coordinated Universal Time” left untranslated.
Dialect / register	109 (2)	[Welsh] llefrith (“milk”, North Wales) vs. llaeth (South Wales); [Maltese] removed the informal filler mela KOLLOX ; [Albanian] passage written in the Geg dialect.

Appendix

Table 5: Types of corrections noted by post-editors, with examples from different locales. We assigned each free-text note to one type by keyword matching (e.g. typo

\to

Spelling, tense

\to

Grammar, register

\to

Dialect). Most notes were too short to classify and fall under Other , so the breakdown is only indicative. The number of notes also varies widely by locale (Maltese post-editors wrote most of them), reflecting differences in annotation habits as much as in translation quality.

Figure 4: Per-language annotator reliability for all three annotators computed using Spearman’s correlation with a leave-one-out strategy, i.e., computing correlation between one annotator and the mean rating from the other two.

	Pointwise (Spearman $\rho$ )					Pairwise (Cohen’s $\kappa$ )
Metric ( $k$ )	Style	Factual	Leaks	Usability	Overall	Style	Factual	Leaks	Usability	Overall
Levenshtein	0.06	0.41	-0.51	–	0.33	0.13	0.75	0.81	–	0.56
Char3-cos	-0.02	0.38	-0.50	–	0.30	0.03	0.66	0.81	–	0.50
Verbatim	0.03	-0.38	0.55	–	0.32	-0.01	0.69	0.87	–	0.52
Length-ratio	0.05	0.17	-0.13	–	0.12	0.10	0.65	0.56	–	0.44
Punct-diff	-0.03	0.21	-0.30	–	0.18	0.07	0.43	0.44	–	0.31

Appendix

Table 6: Pairwise and pointwise statistical diagnostics of evaluation metrics over the four evaluation dimensions and their overall aggregate (mean

\rho

for pointwise, and mean

\kappa

for pairwise). The evaluation is performed on the test split of the dataset, averaged over human annotations from 10 languages. The pointwise alignment is computed with Spearman’s

\rho

, measured against the mean ordinal human ratings for each paraphrase on the respective evaluation dimension. Whereas the pairwise alignment is measured with Cohen’s

\kappa

, against the sign of the difference in ordinal ratings for the two paraphrases, dropping the instances where the difference is zero.

Figure 5: Per-language Spearman’s $\rho$ for the pointwise evaluation of the top-2 overall best-performing metrics for each paradigm for each evaluation dimension.

Figure 6: The aggregate

\Delta R^{2}_{p|.}

(Eq. 1 ) for all evaluation metrics over the four evaluation dimensions against the rest of the evaluation metrics (excluding style embeddings due to their limited language).

Figure 7: Average $\Delta R^{2}_{p\mid q}$ (Eq. 1 ) over different evaluation metric paradigms. Each entry $(x,y)$ denotes the average $\frac{1}{|x|+|y|}\sum_{x_{i}}\sum_{y_{j}}\Delta R^{2}_{x_{i}\mid y_{j}}$ for evaluation paradigms $x$ and $y$ . The semantic embeddings paradigm has only one metric, so we leave the corresponding cell blank. Similarly, usability is evaluated only with LLM-specific metrics; hence, we leave the other comparisons blank.

Figure 8: $\Delta R^{2}_{p\mid q}$ (Eq. 1 ) for a metric pair $m_{p},m_{q}$ . The brighter the cell, the more information it provides relative to the other metrics in the corresponding evaluation dimension.

Metric ( $k$ )	Style	Factual	Leaks	Usability	Overall
Levenshtein (10)	0.39	0.36	0.63	–	0.46
Char3-cos (10)	0.41	0.34	0.59	–	0.44
Verbatim (10)	0.43	0.32	0.69	–	0.48
Length-ratio (10)	0.03	0.13	0.05	–	0.07
Punct-diff (10)	0.06	0.11	0.15	–	0.11
Entropy-diff (10)	0.13	0.16	0.18	–	0.16

Appendix

Table 7: Meta-evaluation of metrics calibrated on chance prediction (

\Phi=0

) and human ceiling (

\Phi=0

); see Section 5.2 for implementation details. We present the results for individual metrics covering string-based metrics, semantic embeddings, style embeddings, open-weight models, and proprietary models. We also provide the performance of various ensemble metrics at the end (denoted by

\rho

Figure 9: Contribution of each metric to the full ensemble $m_{\text{all}}$ for all four evaluation dimensions on the 10 languages. Each cell is the metric’s share of the ensemble’s out-of-fold permutation importance, computed as the metric’s normalized importance. The importance of a metric is computed as the increase in out-of-fold MAE, computed thrice over all 25 out-folds. Negative importance values were clipped to zero before normalizing to obtain the final contribution of each constituent metric. Gray cells indicate that the metric is unavailable for that language. LLMs with less than half of the valid test set judgments are dropped out (see Table 9 ).

Metric	bn	el	en	fil	fr	fy	ja	ko	sl	sw
	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U	S/F/L/U
Levenshtein	.11/.12/.64/–	.54/.84/.81/–	.00/.07/.00/–	.74/.38/.80/–	.16/.02/.47/–	.44/.25/.75/–	.68/.77/.91/–	.19/.06/.18/–	.44/.63/.91/–	.56/.47/.78/–
Char3-cos	.07/.14/.64/–	.57/.87/.83/–	.00/.03/.00/–	.80/.35/.69/–	.16/.10/.58/–	.83/.23/.56/–	.69/.80/.87/–	.16/.08/.17/–	.30/.34/.79/–	.54/.42/.75/–
Verbatim	.11/.12/.67/–	.65/.95/.89/–	.00/.03/.00/–	.85/.45/.90/–	.16/.04/.74/–	.84/.20/.77/–	.63/.45/.97/–	.17/.13/.30/–	.33/.34/.88/–	.58/.47/.81/–
Length-ratio	.00/.03/.05/–	.00/.29/.01/–	.00/.00/.01/–	.07/.01/.04/–	.00/.04/.05/–	.00/.25/.07/–	.16/.53/.11/–	.05/.02/.03/–	.04/.06/.11/–	.07/.04/.04/–
Punct-diff	.00/.03/.19/–	.13/.19/.19/–	.00/.00/.00/–	.00/.12/.07/–	.00/.00/.16/–	.00/.02/.15/–	.41/.37/.14/–	.01/.08/.09/–	.04/.08/.34/–	.04/.23/.19/–

Appendix

Table 8: Metric quality

\mathbf{\Psi}

per language, for every single metric (top) and each ensemble of metrics (bottom). Each cell lists the scores for style, factual consistency, leakage, and usability (S/F/L/U). A score of 0 means the metric does no better than always predicting the mean label, and 1 means it matches the consensus as closely as a single annotator does; values are clipped to

[0,1]

. ‘–’ marks dimensions a metric cannot score (usability has no reference text) and languages a metric does not cover.

Model	bn	el	en	fil	fr	fy	ja	ko	sl	sw	Overall
Aya-8B	6/53	1/8	1/0	3/13	2/5	1/20	0/19	0/2	1/2	0/15	1/14
Gemma4-E4B	1/1	1/3	0/0	1/2	1/3	1/1	1/0	1/2	1/2	1/2	1/2
Gemma4-12B	0/1	0/1	0/0	0/2	0/3	0/1	0/1	0/2	0/1	0/6	0/2
Gemma4-31B	1/0	0/3	0/0	1/2	0/2	0/2	0/2	0/5	0/2	0/4	0/2
Llama31-8B	80/87	47/16	4/0	52/19	11/5	24/8	36/8	26/7	28/10	23/38	33/20
Llama32-3B	63/31	55/56	11/0	29/62	30/37	63/30	52/16	20/3	32/22	42/53	40/35

Appendix

Table 9: Percentage of LLM-judge predictions without a usable score, per language, pooled over the four evaluation dimensions. A prediction counts as failed if the output could not be parsed, was cut off before the score, or was empty or a refusal. Each cell lists pointwise / pairwise. Pairwise rates are based on far fewer items (20–40 per language and dimension, against about 200 pointwise).

Figure 10: The zero-shot prompt used to translate English text to other languages.

Figure 11: The zero-shot prompt used to generate style-personalized text for all languages.

Figure 12: The zero-shot LLM-as-a-judge prompt used to obtain the pointwise LLM responses.

Figure 13: The zero-shot LLM-as-a-judge prompt used to obtain the pairwise LLM responses.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Aug 8, 2025

Evaluating Style-Personalized Text Generation: Challenges and Directions

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.

Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2

Columbia University, USA · Microsoft, USA · The University of York, UK

LLM-as-a-Judge Language Model Generation Evaluation Personalized Text Generation Automated Evaluation Generative AI Evaluation

Ask PDF

Apr 6, 2026

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

While Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to prompt design, language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose OmniScore, a family of complementary, deterministic learned metrics developed using small size (

<

1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision (

\sim

564k instances, in 107 languages) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in 6 languages. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and datasets can be found at https://huggingface.co/collections/QCRI/omniscore

Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar +1

Qatar Computing Research Institute, HBKU, Qatar · †UPES, India

Language Model Generation Evaluation Multilingual Language Model Evaluation LLM-as-a-Judge

Ask PDF

Apr 29, 2026

Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation grounded in authorship science. We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure. Drawing on three measurement traditions - LUAR, a trained authorship verification model; an LLM-as-judge with decoupled trait matching; and classical function-word stylometrics - we evaluate four inference-time personalization methods across 50 authors and 1,000 generations. The theory-grounded metric, LUAR, provides what ad hoc alternatives cannot: calibrated baselines, with a human ceiling of 0.756 and a cross-author floor of 0.626, that give scores absolute meaning. All methods score below this floor, from 0.484 to 0.508, exposing an authorship gap invisible to uncalibrated metrics. The three metrics produce near-zero pairwise correlations, with absolute r less than 0.07, confirming that without theoretical grounding, metric choice determines conclusions: an LLM judge declares a clear winner while LUAR finds no meaningful differentiation. These findings demonstrate the theory-benchmark cycle in action: authorship theory exposes evaluation failures that ad hoc benchmarks miss.

Yash Ganpat Sawant

Independent AI Researcher

LLM Evaluation LLM Personalization Language Model Generation Evaluation Authorship Attribution

Ask PDF

Abstract

Language Model Generation Evaluation

Explore similar work

Evaluating Style-Personalized Text Generation: Challenges and Directions

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization