cs.CLNov 6, 2026

I play of the cello: Evaluating Constructional Bleeding in Multilingual LLMs with CALQUE

Authors: Toshiki Nakai, Yu-Yin Hsu, Marie-Catherine de Marneffe, Joakim Nivre, Jakob Prange, Nathan Schneider, Wesley Scivetti, Tiago Timponi Torrent, 

Organizations: Leipzig University & ScaDS.AI Dresden/Leipzig · The Hong Kong Polytechnic University · CENTAL, UCLouvain & FNRS · Uppsala University · 

Abstract

Languages use different morphosyntactic strategies to express the same meaning. Speakers of multiple languages are able to apply strategies from one language to another, either by mistake or intentionally, to produce a literal translation, or calque. Considering that large language models (LLMs) have been shown to generalize crosslingually, despite being predominantly trained on English, we investigate whether they overgeneralize constructional strategies from English to other languages. To test this, we create a dataset of 5583 sentence pairs across seven languages, matching literal translations of strategies from English into other languages, and literal translations of the other languages' strategies into English. We then quantify constructional bleeding by measuring how often strategies from English are assigned a higher probability than native strategies. We find that models often fail to assign higher probabilities to native strategies and tend to prefer English-derived strategies even when evaluating sentences in other languages. Our findings highlight the limitations of the current English-dominant training paradigm.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Skill Issue: Are Skills Language-Invariant in LLMs?

    Aug 26, 2026Bobby Cheng, Adam Gaber, Zhengyuan Liu +4Cross-Lingual Consistency in Language ModelsMultilingual Language Model Evaluation