cs.CLOct 26, 2026
SaveThe Lay of the Land in Idiom-Translation Quality Estimation
Organizations: Inria, Paris, France
Abstract
Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whether quality estimation (QE) systems have kept pace and can reliably score idiomatic renderings of these expressions above faulty literal ones remains unclear. We introduce the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum. Using this dataset, we benchmark eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges. We find that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones. Higher parameter counts and newer training sets appear to reduce this bias to an extent; nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones, and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality. Our findings caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism.
Figures & tables
| ID | Lang | English Source | Literal Translation | Idiomatic Translation |
|---|---|---|---|---|
| 1228 | EL | As soon as the bill arrived, he vanished into thin air. | \acctonos \acctonos ς ςµ \acctonos , \acctonos ς ς \acctonos \acctonos . | \acctonos \acctonos ς ςµ \acctonos , \acctonos \acctonos . |
| 2448 | ES | When he offered to work for free, I started to smell a rat. | Cuando se ofreció a trabajar gratis, empecé a oler una rata. | Cuando se ofreció a trabajar gratis, me empezó a oler a chamusquina. |
| 2455 | ES | That name doesn’t ring a bell. | Ese nombre no toca una campana. | Ese nombre no me suena. |
| 2437 | ES | I’m so tired, I’m going to hit the sack early tonight. | Estoy tan cansado, voy a golpear el saco temprano esta noche. | Estoy tan cansado, me voy a meter en el sobre temprano esta noche. |
| 3683 | FR | He will finally pay you back when pigs fly. | Il te remboursera enfin quand les cochons voleront. | Il te remboursera enfin quand les poules auront des dents. |
| 4889 | IT | She always bends over backwards to help her friends. | Si piega sempre all’indietro per aiutare i suoi amici. | Si fa sempre in quattro per aiutare i suoi amici. |
Table 1: Examples of single-sentence base triplets, featuring an English source idiom alongside its highly literal and fluent idiomatic translations.
| ID | Lang | Concatenated Source ( ) | 100% Literal Target | 0% Literal (Full Idiomatic) |
|---|---|---|---|---|
| 2991 | ES | He decided to quit smoking and went cold turkey. Don’t worry about the exam, it’s a piece of cake. If you help me move, I’ll scratch your back later. | Él decidió dejar de fumar y se volvió pavo frío. No te preocupes por el examen, es un trozo de pastel. Si me ayudas a mudarme, rascaré tu espalda más tarde. | Él decidió dejar de fumar y lo dejó de golpe. No te preocupes por el examen, es pan comido. Si me ayudas a mudarme, te devolveré el favor más tarde. |
| 5243 | IT | Please don’t fly off the handle over such a small mistake. I won’t be coming to work today, I’m feeling a bit under the weather. Are your ears burning? We were just talking about you. | Per favore, non volare via dal manico per un errore così piccolo. Oggi non verrò al lavoro, mi sento un po’ sotto il tempo. Ti stanno bruciando le orecchie? Stavamo proprio parlando di te. | Per favore, non perdere le staffe per un errore così piccolo. Oggi non verrò al lavoro, sono un po’ giù di tono. Ti fischiano le orecchie? Stavamo proprio parlando di te. |
| 1609 | EL | Trying to convince him is like beating a dead horse. When the alarm went off, he was running around like a headless chicken. The manager really put me through the wringer today. | ς \acctonos \acctonos ς \acctonos ς \acctonos \acctonos \acctonos \acctonos . \acctonos \acctonos ς ςµ \acctonos , \acctonos \acctonos ς \acctonos \acctonos . \acctonos µ \acctonos µ \acctonos ς \acctonos ς \acctonos ς \acctonos µ. | ς \acctonos \acctonos ς \acctonos \acctonos ς \acctonos \acctonos ´ . \acctonos \acctonos ς ςµ \acctonos , \acctonos \acctonos ς \acctonos ς \acctonos . \acctonos µ \acctonos µ \acctonos ς \acctonos ς \acctonos ς \acctonos µ. |
| 1666 | EL | I appreciate that she does not mince her words. I warned him about the danger, but he couldn’t care less. He will finally pay you back when pigs fly. | µ \acctonos \acctonos \acctonos \acctonos . \acctonos ς \acctonos , \acctonos µ \acctonos ς ς \acctonos \acctonos . \acctonos ς \acctonos ς \acctonos \acctonos \acctonos . | µ \acctonos \acctonos µ \acctonos \acctonos \acctonos \acctonos . \acctonos ς \acctonos , \acctonos µ \acctonos ς \acctonos \acctonos ς. ς \acctonos ς, \acctonos ς \acctonos µ \acctonos . |
| 4239 | FR | I have other fish to fry today. He only cleans his apartment once in a blue moon. I worked my fingers to the bone on this project. | J’ai d’autres poissons à frire, aujourd’hui. Il ne nettoie son appartement qu’une fois dans une lune bleue. J’ai travaillé mes doigts jusqu’à l’os sur ce projet. | J’ai d’autres chats à fouetter, aujourd’hui. Il ne nettoie son appartement que tous les 36 du mois. J’ai travaillé d’arrache-pied sur ce projet. |
Table 2: Examples of augmented multi-sentence segments ( ), demonstrating the combination of base triplets prior to gradient permutation.
| ID | Lang | Source ( ) | 100% Literal | 66% Literal | 33% Literal | 0% Literal (Full Idiomatic) |
|---|---|---|---|---|---|---|
| 5359 | IT | He never had to work hard; he was born with a silver spoon in his mouth. My brother lives in Australia, so I only see him once in a blue moon. I won’t be coming to work today, I’m feeling a bit under the weather. | Non ha mai dovuto lavorare sodo; è nato con un cucchiaio d’argento in bocca. Mio fratello vive in Australia, quindi lo vedo solo una volta in una luna blu. Oggi non verrò al lavoro, mi sento un po’ sotto il tempo. | Non ha mai dovuto lavorare sodo; è nato con la camicia. Mio fratello vive in Australia, quindi lo vedo solo una volta in una luna blu. Oggi non verrò al lavoro, mi sento un po’ sotto il tempo. | Non ha mai dovuto lavorare sodo; è nato con la camicia. Mio fratello vive in Australia, quindi lo vedo solo ogni morte di papa. Oggi non verrò al lavoro, mi sento un po’ sotto il tempo. | Non ha mai dovuto lavorare sodo; è nato con la camicia. Mio fratello vive in Australia, quindi lo vedo solo ogni morte di papa. Oggi non verrò al lavoro, sono un po’ giù di tono. |
| 1653 | EL | With no security at the door, the concert was a free-for-all. He makes a mountain out of a molehill when he faces a problem. Since she got engaged, she has been on cloud nine. | \acctonos ς \acctonos ς \acctonos , ς \acctonos \acctonos \acctonos \acctonos \acctonos . \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos µ \acctonos \acctonos \acctonos µ. \acctonos \acctonos \acctonos ς, \acctonos ς ς \acctonos \acctonos . | \acctonos ς \acctonos ς \acctonos , ς \acctonos \acctonos µ \acctonos ς \acctonos \acctonos ς. \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos µ \acctonos \acctonos \acctonos µ. \acctonos \acctonos \acctonos ς, \acctonos ς ς \acctonos \acctonos . | \acctonos ς \acctonos ς \acctonos , ς \acctonos \acctonos µ \acctonos ς \acctonos \acctonos ς. \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos µ \acctonos \acctonos \acctonos µ. \acctonos \acctonos \acctonos ς, \acctonos ς ς \acctonos . | \acctonos ς \acctonos ς \acctonos , ς \acctonos \acctonos µ \acctonos ς \acctonos \acctonos ς. \acctonos ς µ \acctonos \acctonos \acctonos µ \acctonos \acctonos \acctonos µ. \acctonos \acctonos \acctonos ς, \acctonos ς ς \acctonos . |
| 4013 | FR | He only cleans his apartment once in a blue moon. Don’t worry, they are just creating a storm in a teacup. I know this city like the back of my hand. | Il ne nettoie son appartement qu’une fois dans une lune bleue. Ne t’inquiète pas, ils créent juste une tempête dans une tasse de thé. Je connais cette ville comme le dos de ma main. | Il ne nettoie son appartement qu’une fois dans une lune bleue. Ne t’inquiète pas, ils créent juste une tempête dans une tasse de thé. Je connais cette ville sur le bout des doigts. | Il ne nettoie son appartement que tous les 36 du mois. Ne t’inquiète pas, ils créent juste une tempête dans une tasse de thé. Je connais cette ville sur le bout des doigts. | Il ne nettoie son appartement que tous les 36 du mois. Ne t’inquiète pas, ils en font tout un fromage. Je connais cette ville sur le bout des doigts. |
| 5400 | IT | My brother lives in Australia, so I only see him once in a blue moon. She finally let the cat out of the bag and revealed the secret. Buying a new car right now would cost an arm and a leg. | Mio fratello vive in Australia, quindi lo vedo solo una volta in una luna blu. Alla fine ha lasciato il gatto fuori dalla borsa e ha rivelato il segreto. Comprare una macchina nuova in questo momento costerebbe un braccio e una gamba. | Mio fratello vive in Australia, quindi lo vedo solo ogni morte di papa. Alla fine ha lasciato il gatto fuori dalla borsa e ha rivelato il segreto. Comprare una macchina nuova in questo momento costerebbe un braccio e una gamba. | Mio fratello vive in Australia, quindi lo vedo solo ogni morte di papa. Alla fine ha vuotato il sacco e ha rivelato il segreto. Comprare una macchina nuova in questo momento costerebbe un braccio e una gamba. | Mio fratello vive in Australia, quindi lo vedo solo ogni morte di papa. Alla fine ha vuotato il sacco e ha rivelato il segreto. Comprare una macchina nuova in questo momento costerebbe un occhio della testa. |
| 2949 | ES | He was so annoying, I was glad to see the back of him. You should invest in different companies; don’t put all your eggs in one basket. I believe in being honest and calling a spade a spade. | Él era tan molesto, estaba contento de verle la espalda. Deberías invertir en diferentes empresas; no pongas todos tus huevos en una canasta. Creo en ser honesto y llamar a una pala una pala. | Él era tan molesto, estaba contento de verle la espalda. Deberías invertir en diferentes empresas; no te juegues todo a una sola carta. Creo en ser honesto y llamar a una pala una pala. | Él era tan molesto, me alegré de perderlo de vista. Deberías invertir en diferentes empresas; no pongas todos tus huevos en una canasta. Creo en ser honesto y llamar las cosas por su nombre. | Él era tan molesto, me alegré de perderlo de vista. Deberías invertir en diferentes empresas; no te juegues todo a una sola carta. Creo en ser honesto y llamar las cosas por su nombre. |
| 4004 | FR | He ran as fast as he could. Stop beating around the bush and answer the question. She gave up looking for her keys. | Il a couru aussi vite qu’il le pouvait. Arrête de battre autour du buisson et réponds à la question. Elle a abandonné la recherche de ses clés. | Il a couru aussi vite qu’il le pouvait. Arrête de battre autour du buisson et réponds à la question. Elle a renoncé à chercher ses clés. | Il a couru à toute vitesse. Arrête de battre autour du buisson et réponds à la question. Elle a renoncé à chercher ses clés. | Il a couru à toute vitesse. Arrête de tourner autour du pot et réponds à la question. Elle a renoncé à chercher ses clés. |
Table 3: Examples of the semantic gradient for concatenated segments ( ). The intermediate steps (66% and 33% literal) randomly mix literal and idiomatic sentence-level translations to test granular sensitivity.
| QE System | en–ar | en–el | en–es | en–fr | en–it | en–ja | en–ru | en–tr | en–zh |
|---|---|---|---|---|---|---|---|---|---|
| wmt22_cometkiwi_qe | 1.230 | 1.449 | 1.093 | 1.221 | 1.424 | 1.084 | 1.232 | 1.087 | 1.065 |
| wmt23_cometkiwi_xl | 1.237 | 2.338 | 1.010 | 1.784 | 1.959 | 1.263 | 1.296 | 1.224 | 0.887 |
| wmt23_cometkiwi_xxl | 1.183 | 1.949 | 0.647 | 1.248 | 1.297 | 0.990 | 0.754 | 0.682 | 0.479 |
| xcomet_xl | 1.499 | 2.301 | 1.038 | 1.498 | 1.922 | 1.289 | 1.338 | 1.288 | 0.906 |
| xcomet_xxl | 0.896 | 1.125 | 0.778 | 0.971 | 1.030 | 0.916 | 0.788 | 0.685 | 0.675 |
| metricx_23_qe_large | 1.587 | 1.921 | 1.443 | 1.607 | 2.287 | 1.682 | 1.500 | 1.299 | 1.775 |
Table 4: Macro-preference ratio of QE models (Literal / Idiomatic). Values indicate a literal bias, while values indicate an idiomatic preference. The color scale saturates at 0.85 (blue) and 1.15 (red), as the direction of the model’s preference is more informative than the magnitude of the score difference in binary judgments.
| QE System | ar | el | es | fr | it | ja | ru | tr | zh |
|---|---|---|---|---|---|---|---|---|---|
| wmt22_cometkiwi_qe | -0.642 | -0.796 | -0.249 | -0.548 | -0.774 | -0.339 | -0.586 | -0.441 | -0.071 |
| wmt23_cometkiwi_xl | -0.403 | -0.720 | -0.005 | -0.572 | -0.636 | -0.529 | -0.467 | -0.374 | 0.240 |
| wmt23_cometkiwi_xxl | -0.190 | -0.384 | 0.377 | -0.277 | -0.180 | 0.133 | 0.297 | 0.513 | 0.579 |
| xcomet_xl | -0.482 | -0.589 | 0.006 | -0.460 | -0.647 | -0.491 | -0.477 | -0.373 | 0.136 |
| xcomet_xxl | 0.266 | -0.201 | 0.549 | -0.006 | 0.065 | 0.230 | 0.584 | 0.637 | 0.710 |
| metricx_23_qe_large | -0.524 | -0.776 | -0.457 | -0.606 | -0.902 | -0.722 | -0.587 | -0.526 | -0.694 |
Table 5: Mean Kendall’s rank correlation coefficient demonstrating adherence to the idiomaticity gradient. Values closer to indicate a perfect monotonic preference for idiomaticity over literality. N/A values denote instances where a model collapsed into outputting a single, constant score for all permutations within a segment, rendering rank correlation mathematically undefined.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 1: System-Wide Comparison of Preference Ratio Deltas; blue means row rates idiomatic higher than literal by a larger margin than column, and red the opposite; * denotes p < 0.05 via Stratified Bootstrap).
More in this topic
MT Evaluation
Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.
See all 173 papers