Bilingual speakers show cross-lingual activation during reading, especially for words with shared surface form. Cognates (friends) typically lead to facilitation, whereas interlingual homographs (false friends) cause interference or no effect. We examine whether cross-lingual activation in bilingual language models mirrors these patterns. We train Dutch-English causal Transformers under four vocabulary-sharing conditions that manipulate whether (false) friends receive shared or language-specific embeddings. Using psycholinguistic stimuli from bilingual reading studies, we evaluate the models through surprisal and embedding similarity analyses. The models largely maintain language separation, and cross-lingual effects arise primarily when embeddings are shared. In these cases, both friends and false friends show facilitation relative to controls. Regression analyses reveal that these effects are mainly driven by frequency rather than consistency in form-meaning mapping. Only when just friends share embeddings are the qualitative patterns of bilinguals reproduced. Overall, bilingual language models capture some cross-linguistic activation effects. However, their alignment with human processing seems to critically depend on how lexical overlap is encoded, possibly limiting their explanatory adequacy as models of bilingual reading.
Figures & tables
Figure 1: Vocabulary conditions. Full overlap (A) : All identical surface forms that appear in Dutch and English are shared between the two languages, each represented by a single embedding. Friends Overlap (B) : Only cognates and loan words are shared. False Friends Overlap (C) : Only false friends are shared. Minimal Overlap (D) : Only punctuation and named entities are shared, while other tokens have language-specific embeddings. Across all conditions, punctuation and named entities are shared (denoted as ∗ ).
Figure 2: The frequency of each word in the Dutch and English portion of the LM training data. The frequencies are generally positively correlated: a word form that is frequent in Dutch is frequent in English, but more so for friends than false friends . The plot includes (false) friend words from psycholinguistic stimuli ( Bultena et al., 2014 ; Huisman, 2025 ) as well as other compiled lists and our training data (here labeled as Corpus).
Figure 3: Cosine similarity between Dutch and English across the context and word-only embeddings for training data samples for target words (friends or false friends). A larger similarity means the LM considers Dutch and English embeddings more similar.
Figure 4: Surprisal estimates for friends/controls and false friends/controls from LMs trained under different vocabulary conditions. The symbol ∗ indicates a statistically significant difference ( p<.05 ) between the two word types.
Condition
Vocabulary size
Num. parameters
A
77,369
144,672,000
B
141,877
194,214,144
C
144,170
195,975,168
D
144,681
196,367,616
Table 1: Vocabulary size and number of trainable parameters for the language model trained in each condition.
Figure 5: Training and evaluation loss.
Figure 6: Mean pooling over the embeddings in the preceding tokens withing each sentence ( n=500 ). Mean pooling over these to obtain a context embeddings from English samples μCEN for the token winter . The same was performed over Dutch sentences to obtain μCNL(winter) .
Figure 7: Mean pooling over the token representations of the target word in n English sentences ( n=500 ) to obtain μWEN(winter) . Similarly, n Dutch sentences are used to calculate μWNL(winter) .
A concept can carry different associations across languages, while modern language models learn English alongside many other languages during pretraining. Yet comparisons among existing models cannot easily isolate how any one language changes the way these models represent English concepts because their training corpora, compute, architectures, and random seeds all differ. We study this question through a controlled experiment with 40 matched 310M-parameter decoder-only models that share an architecture, tokenizer, training recipe, and English data source. Each bilingual condition adds one of eight languages, while four experimental comparisons separately account for English exposure, total training, and English-document overlap. We align each model pair using 3,000 common English words, then measure where 1,000 held-out English concepts fall along 50 fixed semantic contrasts, such as red versus white. Across 32 experimental comparisons, English concept positions differ more between bilingual and English-only conditions than between English-only runs with different random seeds. These differences are larger in contextual states than in token embeddings and peak in middle layers. The language learned alongside English can therefore change how a model represents English concepts even when its English input representations are explicitly aligned.
Multilingual language models develop shared cross-lingual representations, and various interpretability methods claim to quantify this sharing. These methods have been developed largely in isolation, and when they disagree, it is unclear whether the disagreement reflects a property of the model or an artifact of the measurement. We compare four sharing metrics (CKA, ANC, GMM dominance per token, and ILO) across 21 base models from five families (125M-14B parameters) and correlate each with cross-lingual transfer on five downstream tasks. We find that the metrics differ in their quantification of cross-lingual sharing in these models and suggest that the disagreement traces to anisotropy, the tendency of representations to cluster in a narrow cone of the embedding space. Only ILO's correlation with cross-lingual transfer (Spearman's ρ=0.90) survives controls for model size, family, and per-task variation. We therefore recommend ILO as the primary sharing metric, to be reported alongside anisotropy diagnostics.
Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units. While such sharing is often useful, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ. We investigate this limitation through cross-lingual homographs and false friends, and examine whether introducing language information earlier in the tokenization process can improve their treatment. We propose a simple tokenizer-level intervention based on language cues: language-specific characters replacing initial characters of shared-vocabulary words, reducing common identity during vocabulary construction. In intrinsic analysis, we find through tokenizer-level statistics that BPE and UnigramLM often treat cross-lingual homographs in a largely language agnostic way, whereas the context-sensitive SaGe tokenizer diverges more strongly; our intervention removes this gap. In downstream English-to-X machine translation, our cues yield modest improvements in several settings, especially under BPE, although the effect is not consistent across all languages and evaluation sets. Overall, the findings suggest that adding lightweight language information at the tokenizer level is a promising direction for further exploration.
Rotem Brillant, Yuval Pinter
Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel