Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

G. L. John Salvin; Swapnil Hingmire

cs.CLOct 6, 2026

Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

Authors: G. L. John Salvin, Swapnil Hingmire

Organizations: Indian Institute of Technology Palakkad · Mehta Family School of Data Science and Artificial Intelligence Indian Institute of Technology Palakkad, Kerala, India

Abstract

COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target. We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed. Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied. We trace the effect to the tokeniser and measure it with three label-free diagnostics. The bias is two faults, not one. Scores from different scripts occupy incompatible ranges, and within a single script the metric orders translations less accurately. No order-preserving transform of the score can repair the second fault. The first is removed exactly by COMET-QN, which maps the score distribution of each (language, script) pair onto a shared reference. Pooled agreement with annotators rises from 0.300 to 0.399, which is what makes scores from different scripts safe to place on one axis, and every within-language ordering is provably preserved. A regressor over parity features recovers a further 17.1% of the lost sensitivity. The remainder belongs to the encoder, and no post-processing can reach it. We therefore recommend publishing the normalised score, the three diagnostics, and the identity of the tokeniser they were computed against, so that a reader can tell how much of a score reflects translation quality and how much reflects the writing system.

Figures & tables

(A) One translation, two scripts
“It was the final match for the All Blacks, who had already won the trophy two weeks ago.”
Condition	Human	COMET
MAL native script	$25/25$	$94.4$
Same text, romanised	$25/25$	$56.2$
(B) One source, two error-free targets
“Hydrogen ions are protons that had their electrons stripped off them.”

Table 1: Two segments from IndicMT Eval. Human columns are MQM annotations. Rewriting the script in (A) changes surface form alone, so the rating transfers unchanged; in (B) a single English source is rendered into two languages and both outputs are independently judged error-free, leaving script as the only source of variation.

Language	COMET				TP			IP
	nat	rom	$\Delta$	$\rho_{\mathrm{nat}}{\to}\rho_{\mathrm{rom}}$	nat	rom	$\Delta$	nat	rom	$\Delta$
GUJ a	$85.70$	$70.61$	$-15.09$	$0.590^{{\dagger}}\!\to\!0.341^{{\dagger}}$	$1.388$	$1.599$	$+15.2\%$	$0.438$	$0.289$	$-34.0\%$
TAM	$84.88$	$77.72$	$-7.15$	$0.629^{{\dagger}}\!\to\!0.240^{{\dagger}}$	$1.319$	$2.347$	$+77.9\%$	$0.545$	$0.174$	$-68.0\%$
MAL	$84.08$	$70.07$	$-14.01$	$0.651^{{\dagger}}\!\to\!0.460^{{\dagger}}$	$1.248$	$1.847$	$+48.0\%$	$0.549$	$0.197$	$-64.1\%$
MAR b	$71.92$	$72.94$	$+1.02$	$0.487^{{\dagger}}\!\to\!0.278^{{\dagger}}$	$1.165$	$1.640$	$+40.7\%$	$0.490$	$0.280$	$-42.9\%$
HIN	$71.75$	$68.32$	$-3.43$	$0.422^{{\dagger}}\!\to\!0.230^{{\dagger}}$	$1.210$	$1.702$	$+40.7\%$	$0.636$	$0.265$	$-58.3\%$

Table 2: Mean COMET, segment-level Spearman agreement with MQM, and the TP–IP inversion, in the native (nat) and romanised (rom) conditions. COMET, TP and IP means use all

1{,}400

segments per language;

\rho

uses the working set. a GUJ shows the largest COMET fall despite the smallest TP inflation, because its native IP is the lowest of the five, which gives it the greatest representational burden (§ 4.2 ). b MAR is the sole language whose mean COMET rises while every other quality signal worsens (§ 5 ).

Segment-level severity– $\Delta$ COMET: $\rho=0.183^{{\dagger}}$ , $N=1{,}258$
Severity	$N$	Mean $\Delta$ COMET
Very Low	69	$-3.72$
Low	86	$-1.90$
Medium	184	$+0.73$
High	258	$+1.08$
Very High	661	$+2.63$

Table 3: Mean

\Delta

COMET (romanised minus native) for MAR by MQM severity. The five buckets total

1{,}258

; the remaining

142

segments are Default and bear no severity rank.

\Delta

COMET needs no human score, so the full

1{,}400

-segment base applies.

Corrector (transform class)	$\|\Delta\|$	$\rho_{S}^{\mathrm{lang}}$	$\rho_{S}^{\mathrm{pool}}$
Raw romanised (identity)	$7.88$	$0.310$	$0.160$
Per-lang mean shift (location)	$0.61$	$0.310$	$0.280$
Per-lang affine / $z$ (loc.+scale)	$0.63$	$0.310$	$0.301$
Per-lang quantile map (monotone)	$0.61$	$0.310$	$0.298$
Per-lang isotonic (monotone)	$0.68$	$0.309$	$0.282$
Native score (ceiling)	$0.00$	$0.564$	$0.488$

Table 4: Rank invariance quantified on a 50/50 within-language split (seed 42), so all values are computed on the held-out half:

|\Delta|

is the mean absolute per-language gap after correction (the whole-working-set raw gap is

8.14

, Figure 1 ),

\rho_{S}^{\mathrm{lang}}

the language-averaged and

\rho_{S}^{\mathrm{pool}}

the pooled agreement with MQM over the romanised cells of that half. Every monotone corrector closes the gap and improves pooled comparability while the language-averaged column remains fixed at

0.310

, exactly as Eq. 2 requires.

Figure 1: COMET-QN across the ten (language, script) cells (boxes: median, IQR, 5th–95th percentiles; $6{,}995$ -segment working set). Left, raw COMET (mean $|$ per-lang. gap $|=8.14$ pts; pooled $\rho_{S}=0.300$ ): native cells, filled, sit consistently above their romanised counterparts, open, by a margin that varies across languages and reverses for MAR, which is problem P1 (§ 6.1 ). Right, after COMET-QN (Eq. 3 ; mean $|$ per-lang. gap $|=0.00$ pts; pooled $\rho_{S}=0.399$ ): all cells share the pooled native reference, so the gap is exactly zero by construction and cross-script comparison becomes meaningful. Pooled Pearson agreement rises from $0.324$ to $0.400$ over the same cells, while within-language agreement is unchanged to four decimals, GUJ standing at $0.5903$ before and after. Within-cell ordering is provably untouched (Eq. 2 ).

Held-out	$\rho_{\mathrm{raw}}$	$\rho_{\mathrm{cal}}$	$\rho_{\mathrm{nat}}$	Recovered
GUJ	$0.341$	$0.437$	$0.590$	$38.6\%$
TAM	$0.240$	$0.291$	$0.629$	$13.0\%$
MAL	$0.460$	$0.524$	$0.651$	$33.5\%$
MAR	$0.278$	$0.236$	$0.487$	$-20.3\%$
HIN	$0.230$	$0.270$	$0.422$	$20.8\%$
Mean				$17.1\%$

Table 5: Parity-feature calibration under leave-one-language-out (segment-level Spearman against MQM,

6{,}995

-segment working set). Recovery is the fraction of the raw-to-native sensitivity gap closed. MAR transfers negatively: its bias runs counter to the four training languages, its P1 gap being sign-reversed (§ 6.1 ), so a model fitted on them mispredicts its residual.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Lang.	N	Def	VL	L	M	H	VH
GUJ	$1{,}400$	$248$	$217$	$326$	$390$	$184$	$35$
TAM	$1{,}400$	$60$	$124$	$151$	$323$	$313$	$429$
MAL	$1{,}400$	$372$	$25$	$162$	$443$	$334$	$64$
MAR	$1{,}400$	$142$	$69$	$86$	$184$	$258$	$661$
HIN	$1{,}400$	$399$	$1$	$7$	$134$	$324$	$535$

Appendix

Table 6: Per-language totals and MQM severity counts; buckets sum to

1{,}400

in every language. Def = Default (error-free); VL/L/M/H/VH = Very Low, Low, Medium, High, Very High. Analyses needing a numeric human score use

1{,}400

(GUJ, HIN),

1{,}399

(TAM, MAL) and

1{,}397

(MAR), giving the

6{,}995

-segment working set (§ 3 ).

Language	COMET	BLEURT	BERTScore	BLEU	ChrF	TER
GUJ ${}_{\text{nat}}$	$\mathbf{+0.590^{{\dagger}}}$	$+0.522^{{\dagger}}$	$+0.438^{{\dagger}}$	$+0.339^{{\dagger}}$	$+0.401^{{\dagger}}$	$-0.327^{{\dagger}}$
GUJ ${}_{\text{rom}}$	$+0.341^{{\dagger}}$	$+0.339^{{\dagger}}$	$+0.375^{{\dagger}}$	$+0.340^{{\dagger}}$	$+0.407^{{\dagger}}$	$-0.333^{{\dagger}}$
TAM ${}_{\text{nat}}$	$\mathbf{+0.629^{{\dagger}}}$	$+0.563^{{\dagger}}$	$+0.459^{{\dagger}}$	$+0.320^{{\dagger}}$	$-0.312^{\lx@sectionsign}$	$+0.410^{\lx@sectionsign}$
TAM ${}_{\text{rom}}$	$+0.240^{{\dagger}}$	$+0.377^{{\dagger}}$	$+0.412^{{\dagger}}$	$+0.286^{{\dagger}}$	$+0.408^{\lx@sectionsign}$	$-0.308^{\lx@sectionsign}$
MAL ${}_{\text{nat}}$	$\mathbf{+0.651^{{\dagger}}}$	$+0.596^{{\dagger}}$	$+0.566^{{\dagger}}$	$+0.470^{{\dagger}}$	$+0.542^{{\dagger}}$	$-0.451^{{\dagger}}$
MAL ${}_{\text{rom}}$	$+0.460^{{\dagger}}$	$+0.443^{{\dagger}}$	$+0.529^{{\dagger}}$	$+0.409^{{\dagger}}$	$+0.549^{{\dagger}}$	$-0.466^{{\dagger}}$

Appendix

Table 7: Spearman agreement between each metric and MQM on the working set (

N=1{,}400

for GUJ and HIN,

1{,}399

for TAM and MAL,

1{,}397

for MAR). Bold marks the strongest

|\rho|

per language. COMET loses

|\Delta\rho|=0.19

0.39

under romanisation while the surface metrics stay flat. TER is negatively oriented. § TAM ChrF and TER reverse sign natively owing to character-

n

-gram fragmentation in the reference.

Segment-level $\mathrm{SBI}\geq 3.0$ proportions
Quantity	Native	Romanised
$F$	$2{,}080.8$	$108.3$
$p$	$<10^{-99}$	$3.5\times 10^{-25}$
$\eta^{2}$	$22.9\%$	$1.5\%$
GUJ	$62.1\%$	$95.7\%$
TAM	$34.9\%$	$97.9\%$

Appendix

Table 8: ANOVA on COMET by script family (

N=7{,}000

; the ANOVA and the SBI proportions use metric-side quantities only, so no rows are dropped, and the same test on the

6{,}995

-segment working set gives

F=2{,}080.2

), the share of segments the

\mathrm{SBI}\geq 3.0

flag catches in each condition, the HIN–GUJ contrast, and the dependent-correlation tests ( Meng et al., 1992 ; Steiger, 1980 ; Deutsch et al., 2021 ) .

\eta^{2}=\mathrm{SS}_{\text{between}}/\mathrm{SS}_{\text{total}}

Metric	Ideal	Role
$\mathrm{TP}=\|t(s_{L})\|/\|t(s_{E})\|$	$1.0$	Fragmentation cost; $>1$ over-fragmentation.
$\mathrm{IP}=\mathrm{NLL}(w_{E})/\mathrm{NLL}(w_{L})$	$\approx 1.0$	Semantic density per token; $\ll 1$ collapse.
$\mathrm{SBI}=\mathbb{E}_{i}[\mathrm{TP}_{i}/\mathrm{IP}_{i}]$	$1.0$	Burden per unit information. Flag $\geq 3.0$ .
$\mathrm{IPI}=\|\mathrm{IP}-1.0\|$	$0.0$	Distance from parity; Parity/Burden/Paradox zones.
$\mathrm{Tax}=\mathrm{LP}\times\mathrm{EP}$	$1.0$	Romanisation overhead; LP, EP defined in § 4.2 .

Appendix

Table 9: The two prior parity measures (TP, IP) and the three diagnostics of § 4.2 .

t(\cdot)

is XLM-RoBERTa’s tokeniser and NLL the total negative log-likelihood under BLOOM-560m. Ideal is the value at English-equivalent parity.

Lang.	Script	Family	$\overline{\mathrm{IP}}_{\mathrm{nat}}$	$\overline{\mathrm{TP}}_{\mathrm{nat}}$	$\mathrm{SBI}_{\mathrm{nat}}$	$\mathrm{IPI}_{\mathrm{nat}}$	$\mathrm{IPI}_{\mathrm{rom}}$	mattr	DV/1k	IP–COMET $r$
SPA	Latin	Romance	$1.009$	$1.090$	$1.12$	$0.009$	n/a	$0.581$	n/a	$-0.021^{\mathrm{ns}}$
DEU	Latin	Germanic	$0.535$	$1.133$	$2.24$	$0.465$	n/a	$0.634$	n/a	$-0.262^{{\dagger}}$
GUJ	Gujarati	IA	$0.438$	$1.388$	$3.45$	$0.562$	$0.711$	$0.752$	$36.7$	$-0.064^{{\dagger}}$
TAM	Tamil	Dravidian	$0.545$	$1.319$	$2.73$	$0.455$	$0.826$	$0.862$	$13.6$	$-0.263^{{\dagger}}$
MAL	Malayalam	Dravidian	$0.549$	$1.248$	$2.62$	$0.451$	$0.803$	$0.897$	$42.5$	$-0.266^{{\dagger}}$
MAR	Devanagari	IA	$0.490$	$1.165$	$2.61$	$0.510$	$0.720$	$0.827$	$24.5$	$-0.216^{{\dagger}}$

Appendix

Table 10: Parity profile for all seven settings. Indic values are from the Translation column of IndicMT Eval ( Sai et al., 2023 ) (

N=1{,}400

per language); ENG-DEU and ENG-SPA are from WMT24 ( Kocmi et al., 2024 ) (

339

sources pooled across MT systems,

6{,}006

rows for ENG-DEU and

4{,}651

for ENG-SPA). Controls appear first, then the fixed language order. DV/1k counts isolated dependent-vowel tokens per

1{,}000

XLM-R tokens. IP–COMET

r

is segment-level Pearson over all

1{,}400

segments per language, requiring no human score. n/a : the Latin-script controls have no romanised condition. IA = Indo-Aryan. GUJ’s IP–COMET correlation is weak but significant (

r=-0.064

p=0.017

Figure 2: IPI zone transitions under romanisation. Bands: Parity ( $<0.35$ ), Burden ( $0.35$ – $0.70$ ), Paradox ( $>0.70$ ). Every Indic language crosses into Paradox; ENG-DEU (Burden) and ENG-SPA (Parity) are Latin-script reference points. The callout marks the romanised Malayalam example of Table 1 (A), at $\mathrm{IPI}=0.803$ and COMET $=56.2$ .

Lang.	LP	EP	Tax	EP%
GUJ	$1.15\times$	$1.52\times$	$1.75\times$	$74.6\%$
TAM	$1.78\times$	$3.13\times$	$5.57\times$	$66.4\%$
MAL	$1.48\times$	$2.78\times$	$4.12\times$	$72.3\%$
MAR	$1.41\times$	$1.75\times$	$2.47\times$	$62.1\%$
HIN	$1.41\times$	$2.40\times$	$3.37\times$	$71.9\%$

Appendix

Table 11: Length and Entropy Penalty components of the Computational Tax, with

\mathrm{EP\%}=\ln(\mathrm{EP})/\ln(\mathrm{Tax})

. The Entropy Penalty dominates in every language, so information lost per token, not tokens gained, is the principal cost of romanisation.

Lang.	Native $\rho$	Romanised $\rho$
GUJ	$0.883^{{\dagger}}$	$0.952^{{\dagger}}$
TAM	$0.850^{{\dagger}}$	$0.905^{{\dagger}}$
MAL	$0.875^{{\dagger}}$	$0.900^{{\dagger}}$
MAR	$0.891^{{\dagger}}$	$0.942^{{\dagger}}$
HIN	$0.897^{{\dagger}}$	$0.956^{{\dagger}}$

Appendix

Table 12: Segment-level Spearman correlation between word count and XLM-R token count (all

p<10^{-99}

N=1{,}400

). Romanisation drives these toward

1.0

, the signature of single-character fallback.

Setting	$\overline{\mathrm{IP}}$	SBI	MATTR	Byte pr.	1-tok%	IP–COMET $r$
ENG-SPA (SPA)	$1.009$	$1.12$	$0.581$	$1.09\times$	$22.1\%$	$-0.021^{\mathrm{ns}}$
ENG-DEU (DEU)	$0.535$	$2.24$	$0.634$	$1.25\times$	$14.6\%$	$-0.262^{{\dagger}}$
Indic nat.	$0.438$ – $0.636$	$2.03$ – $3.45$	$0.633$ – $0.897$	$2.45$ – $5.33\times$	$8.7$ – $30.3\%$	varies

Appendix

Table 13: Latin-script controls beside the Indic native range. 1-tok% is the share of unique word types receiving a single-token XLM-R encoding. Zones: ENG-SPA Parity; ENG-DEU Burden; Indic native Burden; Indic romanised Paradox.

Lang.	Script	MATTR nat	MATTR rom	Byte Prem.
GUJ	Gujarati	$0.752$	$0.665$ $\downarrow$	$3.03$
TAM	Tamil	$0.862$	$0.733$ $\downarrow$	$5.30$
MAL	Malayalam	$0.897$	$0.768$ $\downarrow$	$5.33$
MAR	Devanagari	$0.827$	$0.721$ $\downarrow$	$3.64$
HIN	Devanagari	$0.633$	$0.601$ $\downarrow$	$2.45$
SPA	Latin (Romance)	$0.581$	n/a	$1.09$

Appendix

Table 14: MATTR and Byte Premium across all seven settings as vocabulary-free and encoding-independent checks on the TP and IP findings. MATTR uses

w=500

with whitespace tokenisation ( Covington and McFall, 2010 ) ; Byte Premium is the target-to-English ratio of UTF-8 bytes per word ( Arnett et al., 2024 ; Arnett and Bergen, 2025 ) .

\downarrow

marks a fall under romanisation; n/a : no romanised condition exists for the Latin-script controls.

Raw	Rank	Position	Mapped
$71.07$	$1$	$0.167$	$68.21$
$76.41$	$2$	$0.333$	$77.27$
$78.14$	$3$	$0.500$	$83.14$
$78.24$	$4$	$0.667$	$87.58$
$80.33$	$5$	$0.833$	$91.48$

Appendix

Table 15: Eq. 3 applied to five romanised GUJ segments. Position is

\mathrm{rank}/(n{+}1)

; Mapped is the value at that position of the pooled native-script reference. Ranks are carried through unchanged, which is why the operation cannot disturb any within-cell ordering.

Explore similar work

Aug 17, 2026cs.CL

IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020-2024 shared-task lineage with an extended English-Malayalam resource into IndicQE-APE:

126{,}754

instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level tags and an error explanation, and a test set stratified over four difficulty axes. We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on direct assessment and select a compressed slice of it. Segments whose segment-level and token-level signals disagree are ranked below equally scored segments of the same language. Four-shot prompting costs every model at or below

3.4

B both correlation and output-format compliance. Unedited MT beats every APE system we run on three of the four pairs. The benchmark (https://huggingface.co/datasets/surrey-nlp/IndicQE-APE) and code (https://github.com/surrey-nlp/IndicQE-APE) are released.

Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare +14

University of Surrey · 2IIT Bombay · University of Oslo +9

Language Pairs Image Quality Assessment Lineages

Ask PDF

Jun 1, 2026cs.CL

SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation

Word Error Rate (WER) is the dominant metric for automatic speech recognition (ASR), but it can overestimate errors when references and hypotheses encode the same words in different scripts. This issue is common in multilingual settings where ASR models may emit romanized text. We propose Script-Normalized WER (SN-WER), a training-free, evaluation-only scoring method that transliterates both reference and hypothesis text into a language-specific canonical script before computing WER. We evaluate SN-WER on 5 Indic languages, 2 datasets, and 3 ASR models. On curated FLEURS data, SN-WER reduces inflated model gaps by up to 12%, while on noisier Common Voice data the reductions are smaller or inconsistent, indicating genuine recognition weaknesses rather than only script mismatch. Controlled stress tests show a 67% attenuation of artificial romanization-induced WER inflation, while lexical-substitution controls show near-identical sensitivity to semantic errors, with Delta SN-WER / Delta WER approximately 1.09. SN-WER is robust to transliterator choice, normalization changes, and shows low token-collision rates below 0.1% in the evaluated Indic setting. We argue that SN-WER should be reported alongside WER and CER as a companion metric for script-insensitive ASR evaluation, especially when transcripts feed downstream search, indexing, or multilingual LLM pipelines.

Priyaranjan Pattnayak

Oracle America Inc.

Automatic Speech Recognition Evaluation Character Error Rate Automatic Speech Recognition

Ask PDF

Apr 20, 2026cs.CL

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Machine Translation Quality Machine Translation Multilingual Benchmark Linguistics

Ask PDF