While monolingual language model (LM) surprisal is widely established as a predictor of reading times, it cannot capture the non-native (L2) reading dynamics that are shaped by the reader's first language (L1). In this work, we investigate whether the cross-lingual transfer within bilingual models aligns with human L1-driven biases during L2 reading. We evaluate bilingual models on eye-tracking data of readers from 12 diverse L1 backgrounds reading English texts. To isolate L1-specific transfer, we compare bilingual models trained on an L1 that matches the readers' L1 background to both a monolingual English baseline and bilingual models trained on other L1s. We demonstrate that matching the readers' L1 improves the prediction of L2 reading times. This predictive advantage is concentrated in late-stage eye-tracking measures, modulated by L1-L2 script overlap, and peaks among intermediate-proficiency readers. These findings provide evidence that bilingual LMs capture aspects of cross-linguistic influence, and that such influence is present in L2 reading behavior.
Figures & tables
Figure 1: Difference in explained variance ΔRpseudo2 (scaled by 104 ) in four reading measures as a function of SoE. The blue line represents the mean across all L2 reader subsets, while the red line denotes the native English (En-UK) control group. Faint lines indicate individual model evaluations. Positive values indicate a predictive advantage for the bilingual surprisal. With the exception of first fixation duration, bilingual surprisal consistently provides greater predictive power for L2 than L1 readers.
Figure 2: Per-word Δ LL (scaled by 104 ) between matched and mismatched models in four reading measures as a function of SoE. Solid lines and shaded bands represent subset means with 95% confidence intervals for the bilingual models (blue) and monolingual baseline (red), with faint blue lines indicating individual subset evaluations. Higher values indicate a stronger predictive advantage for the matched model relative the mismatched one. With the exception of first fixation duration, bilingual models match or outperform the monolingual baseline.
Metric
Mean ΔLL
Sig. Ratio ( p<0.05 )
Match
Mism.
Match
Mism.
First Fixation
1.07
1.33
17.0%
23.2%
Gaze Duration
3.23
1.78
55.0%
30.4%
Go-Past
3.32
1.69
53.7%
30.8%
Total Fixation
6.53
4.00
77.3%
64.3%
Table 1: Nested model comparisons ( mboth ) evaluating the unique contribution of matched versus mismatched bilingual surprisal averaged across all SoE models and language subsets. With the exception of First Fixation, matched models’ mean ΔLL and significance ratio are higher than the mismatched models.
Figure 3: Per-word Δ LL (scaled by 104 ) between matched and mismatched models in four reading measures as a function of SoE, grouped by Latin (top) and Non-Latin (bottom) L1 scripts. Solid lines and shaded bands denote subset means with 95% confidence intervals for the bilingual models (blue) and monolingual baseline (red). Faint blue lines indicate individual subset traces. Early measures (First Fixation and Gaze Duration) reveal a clear divergence in predictive behavior between the two script groups.
Figure 4: Bilingual surprisal advantage ΔR2 (scaled by 104 ) across readers’ English proficiency (LexTALE %) in four reading measures. Curves show quadratic fits per SoE model; circular markers denote binned means ( ±95% CI) at SoE=48k. Faint dots indicate individual observations, and the horizontal dotted line denotes zero gain. Inset labels indicate the significance of the quadratic proficiency term in a WLS with L1-subset fixed effects and standard errors clustered by L1 subset ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 , nsnot significant ). While early measures show negligible non-linearity, late measures (Go-Past, Total Fixation) show an inverted-U trajectory peaking among intermediate-proficiency readers ( 65–80% ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Subset
Language & Site
Samples
Readers
hi_iiith
Hindi (Hyderabad)*
77,522
53
en_uk
English (UK)
66,041
50
bp
Brazilian Portuguese
62,608
54
ge_zu
German (Zurich)
58,949
47
hi_iitk
Hindi (Kanpur)*
58,642
45
ru_mo
Russian (Moscow)*
57,817
49
Appendix
Table 2: MECO L2 subsets, corresponding L1 languages and recording site (* denotes Non-Latin), total number of samples (before filtering) and number of readers.
Hyperparameter
Value
Layers
12
Embedding size
768
Hidden size
768
Intermediate hidden size
3072
Attention heads
12
Attention head size
64
Appendix
Table 3: Architectural and optimization hyperparameters for the GPT-2 bilingual models.
Figure 5: Pairwise Pearson correlation of English surprisal values between all L1 bilingual models across the SoE stages. The dark blue line represents the mean correlation, the shaded band shows 95% confidence interval, and the light blue lines show individual pairwise comparisons.
Figure 6: Explanatory power ( ΔR2 over the baseline, scaled by 104 ) of monolingual and of bilingual surprisal across readers’ English proficiency (LexTALE %) in four reading measures. Faint dots indicate individual observations, and the horizontal dotted line denotes zero gain. Inset labels indicate the significance of the quadratic proficiency term in a WLS with L1-subset fixed effects and standard errors clustered by L1 subset ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 , nsnot significant ). Both bilingual and monolingual models exhibit a pronounced inverted-U relationship with proficiency.
Metric
SoE
N
βquad
pquad
ΔAIC
First Fixation
0
480
+0.002
0.464
− 1.5
16k
480
+0.002
0.494
− 1.7
32k
480
− 0.001
0.793
− 2.0
48k
480
+0.003
0.436
− 1.5
Gaze Duration
0
480
− 0.007
0.129
− 0.2
16k
480
− 0.007
0.093
+0.0
Appendix
Table 4: Model comparison evaluating non-linear proficiency moderation across eye-tracking measures. βquad denotes the estimated coefficient of the quadratic proficiency term, and pquad denotes its robust HC3 significance. ΔAIC=AIClinear−AICquadratic , where values >+2.0 indicate that the quadratic model improves on the linear baseline by more than the penalty for its extra parameter ( Burnham and Anderson, 2002 ) . Boldface marks exposure stages where both criteria favor the quadratic model ( pquad<0.05 , ΔAIC>2.0 ).
Figure 7: Bilingual surprisal advantage ( ΔR2=Rbilingual2−Rmonolingual2 , scaled by 104 ) modeled against composite L2 proficiency (PC1 extracted across LexTALE, Vocabulary, Spelling, TOWRE word, and TOWRE nonword; with 52.5% variance explained) in four reading measures. Curves depict second-order polynomial regressions across SoEs; markers denote empirical binned means ( ±95% CI) at SoE=48k. Inset labels indicate the significance of the quadratic proficiency term in a WLS with L1-subset fixed effects and standard errors clustered by L1 subset ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 , nsnot significant ). Faint points represent individual participants, and the dotted horizontal line marks zero gain.
Bilingual speakers show cross-lingual activation during reading, especially for words with shared surface form. Cognates (friends) typically lead to facilitation, whereas interlingual homographs (false friends) cause interference or no effect. We examine whether cross-lingual activation in bilingual language models mirrors these patterns. We train Dutch-English causal Transformers under four vocabulary-sharing conditions that manipulate whether (false) friends receive shared or language-specific embeddings. Using psycholinguistic stimuli from bilingual reading studies, we evaluate the models through surprisal and embedding similarity analyses. The models largely maintain language separation, and cross-lingual effects arise primarily when embeddings are shared. In these cases, both friends and false friends show facilitation relative to controls. Regression analyses reveal that these effects are mainly driven by frequency rather than consistency in form-meaning mapping. Only when just friends share embeddings are the qualitative patterns of bilinguals reproduced. Overall, bilingual language models capture some cross-linguistic activation effects. However, their alignment with human processing seems to critically depend on how lexical overlap is encoded, possibly limiting their explanatory adequacy as models of bilingual reading.
Iza Škrjanec, Irene Elisabeth Winther, Vera Demberg +1
Saarland University, Germany · Zuse School ELIZA, Germany · Radboud University, the Netherlands
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled language model pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable, enabling systematic and comparable experimentation of training conditions. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s) to study multilingual pretraining and computational modelling of bilingualism and second language learning. Evaluating models on human bilingual and second language reading-time prediction and grammaticality judgement tasks, we find that staged and temporally structured curricula consistently improve alignment with language learner reading time compared to balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs. The Beetle models are well suited tools to help move computational psycholinguistics beyond its prevailing monolingual, English-centric focus toward models of human bilingual processing, to study cross-lingual learning dynamics, while supporting community-based development of controlled model families.
Suchir Salhan, Catherine Arnett, James Michaelov +1
Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.
Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4
ETH Zürich · Toyota Technological Institute at Chicago · University College London