Scaling Multilingual Pretraining of Pixel Language Models

Ilker Kesen; Jenny Kunz; Kasra Malihi; Desmond Elliott

cs.CLOct 21, 2026

Scaling Multilingual Pretraining of Pixel Language Models

Authors: Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott

Organizations: Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen

Abstract

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.

Figures & tables

Figure 1: mpixel performance when scaling the number of pretraining languages and model capacity. Adding new languages steadily improves performance on seen scripts, albeit with a slight decrease on shared languages. Higher capacity improves performance on shared and seen scripts. On unseen scripts, neither additional languages nor capacity yields consistent gains.

	$\theta$	L	Patches	Words
mpixel	86M	16	120B	38B
mpixel large	307M	16	120B	38B
mpixel 4s	86M	4	30B	10B
mpixel 8l	86M	8	60B	19B
mpixel 8s	86M	8	60B	19B

Table 1: Overview of the trained models. The mpixel and mpixel large models as the main outcomes of this paper, while the 4s , 8l and 8s models are used in our control experiments.

\theta

: number of parameters. L: number of pretraining languages. Patches: number of unique image patches seen during pretraining. Words: approximate number of white-space delimited words.

Model	$\|\theta\|$	#L	$\|X\|$	SIB-200 (77)	NER (12)	UDP (30)	XNLI (15)	GLUE (1)	Avg
canine-s	127M	104	1.0T	55.8	70.8	69.7	70.5	73.0	68.0
mbert	179M	104	-	63.0	81.6	80.0	74.7	79.6	75.8
xlm-r	270M	100	6.3T	65.4	83.1	82.7	79.1	80.8	78.2
pixel $\dagger$	86M	1	0.1T	52.9	73.6	75.7	67.1	78.7	69.6
mpixel	86M	16	0.5T	61.9	77.3	80.3	69.5	73.3	72.5
mpixel	307M	16	0.5T	64.4	80.9	80.7	75.0	79.6	76.1

Table 2: Performance of mpixel compared to subword vocabulary and character-level models. Each cell is the average over all evaluation languages of the best three runs per language, selected by validation performance. We report F1 for SIB-200 and NER, LAS for UDP, accuracy for XNLI and the average score for GLUE, over (

\cdot

) languages.

|\theta|

: model size. #L: number of pretraining languages.

|X|

: pretraining size in characters for canine-s , subwords for xlm-r , and image patches for the mpixel models. Best results are in bold , second best underlined .

Figure 2: Performance of models on typologically distant low-resource languages in the SIB-200 benchmark.

Figure 3: Downstream performance vs. number of pretraining languages (86M models;

x{=}1

is English-only). Lines are evaluation tiers fixed w.r.t. mpixel ’s languages and scripts; each point is a mean over that tier’s items (

n

per panel, in legend order). At

x{=}8

the point is the mean of mpixel 8l and mpixel 8s and the bar spans them.

Figure 4: Performances of the mpixel variants on the downstream benchmarks considering only four pretraining languages shared across models.

Figure 5: ABX probing performances of mpixel variants on the four shared pretraining languages.

Figure 6: Layer-wise probing accuracy on sentence-level linguistic tasks. The accuracy is plotted against the proportional encoder layer depth for both base and large model variants.

Figure 7: Layer-wise probing accuracy on linspector word-level tasks. Rows show tasks, columns show languages; each subplot plots accuracy against encoder layer.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Script	Pivot	Distance
arb	Latin	hau	0.714
ayr	Latin	swh	0.725
bam	Latin	jav	0.609
cjk	Latin	swh	0.555
dyu	Latin	zsm	0.623
fij	Latin	sun	0.603

Appendix

Table 3: Language–script pairs selected as typologically distant (DistaLs average of typology-related distances,

T=0.50

). The nearest same-script pretraining pivot and distance are shown.

ISO	Language	Script	Language Family	Branch	# Docs (M)	# Words (B)	# Chars (B)
ben	Bengali	Bengali	Indo-European	Indo-Aryan	6.9	2.5	13.9
dan	Danish	Latin	Indo-European	Germanic	16.7	2.3	12.2
eng	English	Latin	Indo-European	Germanic	17.0	2.5	12.2
fin	Finnish	Latin	Uralic	Finnic	16.4	1.8	13.0
heb	Hebrew	Hebrew	Afro-Asiatic	Semitic	17.0	2.5	11.7
hin	Hindi	Devanagari	Indo-European	Indo-Aryan	18.6	3.7	15.0

Appendix

Table 4: Per-language statistics of the pretraining corpus: script, language family and branch, and corpus size in documents (M, millions), whitespace-delimited words (B, billions) and characters (B). Word counts are omitted for languages without whitespace word boundaries (Chinese, Thai).

Language	Treebank	#Sentences	License
eng	English-EWT	16621	CC BY-SA 4.0
ara	Arabic-PADT	7664	CC BY-NC-SA 3.0
hye	Armenian-BSUT	2523	CC BY-SA 4.0
bul	Bulgarian-BTB	11138	CC BY-NC-SA 3.0
cop	Coptic-Scriptorium	2011	CC BY 4.0
dan	Danish-DDT	5512	CC BY-SA 4.0

Appendix

Table 5: Total number of sentences of Universal Dependencies v2.18 ( Zeman et al., 2026 ; Nivre et al., 2020 ) treebanks used for dependency parsing task evaluations, including dataset licenses. Adapted from Rust et al. (2023) ; Kesen et al. (2025) .

Language	Train	Validation	Test	Source	License
ben	961679	4859	607	Naamapadam	CC0 1.0
dan	4383	564	565	Universal NER (UNER_Danish-DDT)	CC BY-SA 4.0
ell	1662	403	456	Universal NER (UNER_Greek_GDT)	CC-BY-SA-4.0
eng	12543	2001	2077	Universal NER (UNER_English-EWT)	CC BY-SA 4.0
heb	5241	484	491	Universal NER (UNER_Hebrew-HTB)	CC BY-SA 4.0
hin	985787	13460	867	Naamapadam	CC0 1.0

Appendix

Table 6: Data distribution and license information for the NER datasets used within this work. See Universal NER ( Mayhew et al., 2024 ; Blevins et al., 2026 ) , Naamapadam ( Mhaske et al., 2023 ) , NER-UK ( Chaplynskyi and Romanyshyn, 2024 ) and KLUE ( Park et al., 2021 ) for more detailed statistics.

Parameter	Value
Image size	(16, 8192, 3)
Patch size $P$	16
Layer norm $\varepsilon$ ( Ba et al., 2016 )	$1\text{\times}{10}^{-12}$
Span masking ratio $R$	0.25
Span masking max length $S$	6
Span masking cumulative weights $W$	$\{0.2,0.4,0.6,0.8,0.9,1\}$

Appendix

Table 7: Pretraining hyperparameters shared by all mpixel models and pixel

\dagger

Parameter	Base Models	mpixel large
Encoder hidden size $D_{\text{enc}}$	768	1024
Encoder intermediate size	3072	4096
Encoder num attention heads	12	16
Encoder num layers $L$	12	24
Decoder hidden size $D_{\text{dec}}$	512	512
Decoder intermediate size	2048	2048

Appendix

Table 8: Pretraining hyperparameters that differ between base-scale models with 86M parameters and the large mpixel model with 307M parameters.

Parameter	SIB-200	UDP	NER	XNLI	GLUE
Classification head pooling	Mean	—	—	CLS	Mean
Optimizer	AdamW
Adam $\beta$	0.9, 0.999
Adam $\epsilon$	1e $-$ 8
Weight decay	0
Learning rate schedule	Linear decay

Appendix

Table 9: Hyperparameters used for fine-tuning and evaluating models on SIB-200, UDP, NER, XNLI and GLUE.

Language	Script	canine-s	mbert	xlm-r	pixel $\dagger$	mpixel 4s	mpixel 8l	mpixel 8s	mpixel	mpixel
Language	Script	127M	179M	270M	86M	86M	86M	86M	86M	307M
Pretraining Languages
arb	Arab	69.2 $\pm$ 4.0	84.1 $\pm$ 2.3	88.2 $\pm$ 0.7	42.7 $\pm$ 3.2	48.2 $\pm$ 1.6	47.6 $\pm$ 4.5	73.8 $\pm$ 3.1	75.3 $\pm$ 2.6	77.5 $\pm$ 1.9
ben	Beng	54.5	78.5 $\pm$ 1.6	83.1 $\pm$ 1.2	48.6 $\pm$ 0.4	58.4 $\pm$ 1.9	55.7 $\pm$ 1.2	58.0 $\pm$ 3.1	78.2 $\pm$ 2.3	79.2 $\pm$ 1.2
dan	Latn	72.7 $\pm$ 1.8	83.4 $\pm$ 0.6	88.0 $\pm$ 0.7	67.3 $\pm$ 0.6	60.8 $\pm$ 1.4	81.4 $\pm$ 2.0	61.9 $\pm$ 1.4	82.7 $\pm$ 0.8	84.1 $\pm$ 0.6
ell	Grek	65.3	74.0 $\pm$ 1.8	85.4 $\pm$ 1.3	42.0 $\pm$ 1.3	49.6 $\pm$ 0.3	50.0 $\pm$ 1.5	84.3 $\pm$ 1.7	82.3 $\pm$ 1.8	81.5 $\pm$ 1.2

Appendix

Table 10: SIB-200 per-language test F1. Scores are averaged over the 3 runs with the best validation F1. canine-s scores given without a standard deviation are imported from Kesen et al. (2025) . xlm-r results are taken from Adelani et al. (2024) .

	canine-s	mbert	xlm-r	pixel $\dagger$	mpixel 4s	mpixel 8l	mpixel 8s	mpixel	mpixel
	127M	179M	270M	86M	86M	86M	86M	86M	307M
Pretraining Languages
eng	67.1	81.5 $\pm$ 1.1	83.2 $\pm$ 0.7	81.1 $\pm$ 0.8	78.9 $\pm$ 0.2	77.0 $\pm$ 0.5	75.5 $\pm$ 0.8	74.9 $\pm$ 1.8	80.9 $\pm$ 0.7
dan	70.5 $\pm$ 0.4	86.5 $\pm$ 0.2	86.6 $\pm$ 0.6	76.3 $\pm$ 1.1	74.1 $\pm$ 0.3	80.7 $\pm$ 1.6	74.3 $\pm$ 0.1	78.4 $\pm$ 2.1	85.8 $\pm$ 0.2
ell	56.1 $\pm$ 1.5	78.8 $\pm$ 0.4	81.1 $\pm$ 0.6	60.9 $\pm$ 0.8	62.5 $\pm$ 0.8	62.9 $\pm$ 1.4	75.5 $\pm$ 1.0	70.0 $\pm$ 0.5	79.8 $\pm$ 0.2
heb	42.0 $\pm$ 1.5	73.4 $\pm$ 0.7	78.4 $\pm$ 0.7	57.1 $\pm$ 1.0	59.5 $\pm$ 1.7	58.0 $\pm$ 1.0	58.3 $\pm$ 1.4	68.2 $\pm$ 1.0	72.9 $\pm$ 0.6

Appendix

Table 11: NER per-language test F1. Validation split F1 is reported for kor , as it does not include a test set. Scores are averaged over the 3 runs with the best validation F1. CANINE-S scores given without a standard deviation are imported from Kesen et al. (2025) .

Model	$\theta$	#L	arb	bul	deu	ell	eng	spa	fra	hin	rus	swa	tha	tur	urd	vie	zho	Avg
canine-s	127M	104	69.7	75.0	74.9	72.5	79.1	76.3	75.3	65.2	73.0	65.0	62.3	68.9	64.1	71.3	65.6	70.5
mbert	179M	104	73.2	77.7	77.5	75.7	83.5	79.8	78.6	70.1	76.4	68.1	67.2	73.8	64.4	76.5	77.9	74.7
xlm-r	270M	100	77.3	81.3	80.3	80.4	85.4	82.2	81.4	76.1	79.7	73.1	77.9	78.6	73.0	79.7	80.2	79.1
pixel $\dagger$	86M	1	60.3 $\pm$ 0.4	69.4 $\pm$ 0.3	72.4 $\pm$ 0.1	69.2 $\pm$ 0.4	83.0 $\pm$ 0.4	74.6 $\pm$ 0.6	75.0 $\pm$ 0.4	61.6 $\pm$ 0.2	66.5 $\pm$ 0.4	63.0 $\pm$ 0.3	51.5 $\pm$ 0.3	66.2 $\pm$ 0.4	57.1 $\pm$ 0.2	69.4 $\pm$ 0.3	67.1 $\pm$ 0.5	67.1
mpixel 4s	86M	4	61.3 $\pm$ 0.4	74.8 $\pm$ 0.3	71.8 $\pm$ 0.5	71.4 $\pm$ 0.1	79.8 $\pm$ 0.2	74.2 $\pm$ 0.2	73.7 $\pm$ 0.2	70.6 $\pm$ 0.4	73.0 $\pm$ 0.1	63.0 $\pm$ 0.7	54.4 $\pm$ 0.6	66.5 $\pm$ 0.1	58.3 $\pm$ 0.4	69.9 $\pm$ 0.1	75.9 $\pm$ 0.0	69.2
mpixel 8l	86M	8	58.4 $\pm$ 0.3	72.9 $\pm$ 0.2	71.5 $\pm$ 0.1	67.9 $\pm$ 0.2	79.7 $\pm$ 0.3	77.5 $\pm$ 0.2	73.2 $\pm$ 0.2	69.8 $\pm$ 0.1	71.7 $\pm$ 0.4	60.9 $\pm$ 0.3	51.3 $\pm$ 0.5	74.3 $\pm$ 0.1	55.9 $\pm$ 0.0	67.5 $\pm$ 0.2	74.6 $\pm$ 0.1	68.5

Appendix

Table 12: XNLI per-language test accuracy. Scores are averaged over the 3 runs with the best validation accuracy. xlm-r scores are imported from Conneau et al. (2020) , and canine-s and mbert scores from Sun et al. (2023) .

Model

\theta

MNLI-M/MM

QQP

QNLI

SST-2

CoLA

STS-B

MRPC

RTE

WNLI

Avg

393k

364k

105k

67k

8.6k

5.8k

3.7k

2.5k

635

canine-s

127M

104

77.3

\pm

0.1 / 78.0

\pm

0.1

85.3

\pm

0.2

88.8

\pm

0.2

88.0

\pm

0.0

17.4

\pm

0.6

85.1

\pm

0.0

88.3

\pm

0.3

65.0

\pm

0.6

56.8

\pm

0.7

73.0

mbert

179M

104

82.3

\pm

0.1 / 82.8

\pm

0.1

87.3

\pm

0.0

91.1

\pm

0.2

91.1

\pm

0.2

46.1

\pm

1.1

89.2

\pm

0.0

90.8

\pm

0.1

76.7

\pm

0.9

58.7

\pm

1.8

79.6

xlm-r

270M

100

84.7

\pm

0.1 / 84.7

\pm

0.1

88.0

\pm

0.1

90.6

\pm

0.0

93.1

\pm

0.1

60.3

\pm

0.2

88.8

\pm

0.1

92.0

\pm

0.1

69.0

\pm

0.3

56.3

\pm

0.0

80.8

pixel

\dagger

86M

83.2

\pm

0.1 / 83.6

\pm

0.3

87.5

\pm

0.2

89.6

\pm

0.1

91.9

\pm

0.1

49.4

\pm

0.4

88.5

\pm

0.0

91.4

\pm

0.2

65.6

\pm

0.5

56.8

\pm

1.3

78.7

mpixel 4s

86M

80.5

\pm

0.2 / 80.7

\pm

0.1

87.4

\pm

0.1

88.2

\pm

0.1

90.0

\pm

0.2

37.5

\pm

0.3

86.2

\pm

0.3

90.1

\pm

0.4

63.4

\pm

0.9

57.8

\pm

1.1

76.2

Appendix

Table 13: GLUE test results. Scores are averaged over the 3 runs with the best validation score.

Language	Script	canine-s	mbert	xlm-r	pixel $\dagger$	mpixel 4s	mpixel 8l	mpixel 8s	mpixel	mpixel
$\|\theta\|$		127M	179M	270M	86M	86M	86M	86M	86M	307M
Pretraining Languages
arb	Arab	77.5	85.5 $\pm$ 0.1	86.6 $\pm$ 0.1	80.0 $\pm$ 0.0	79.4 $\pm$ 0.1	79.8 $\pm$ 0.0	82.8 $\pm$ 0.2	82.2 $\pm$ 0.1	83.2 $\pm$ 0.3
zho	Hant	75.1	85.4 $\pm$ 0.1	86.5 $\pm$ 0.3	71.2 $\pm$ 0.2	83.0 $\pm$ 0.4	82.6 $\pm$ 0.5	82.4 $\pm$ 0.2	81.6 $\pm$ 0.3	85.0 $\pm$ 0.2
dan	Latn	68.0 $\pm$ 0.7	87.5 $\pm$ 0.1	89.8 $\pm$ 0.4	80.3 $\pm$ 0.1	80.2 $\pm$ 0.3	86.6 $\pm$ 0.1	80.3 $\pm$ 0.2	85.7 $\pm$ 0.3	88.2 $\pm$ 0.1
eng	Latn	82.4	91.8 $\pm$ 0.2	93.2 $\pm$ 0.1	92.5 $\pm$ 0.1	89.3 $\pm$ 0.2	89.2 $\pm$ 0.1	88.7 $\pm$ 0.1	88.7 $\pm$ 0.2	90.5 $\pm$ 0.1

Appendix

Table 14: UDP per-language test LAS. Scores are averaged over the 3 runs with the best validation LAS. canine-s scores given without a standard deviation are imported from Kesen et al. (2025) .

Figure 8: ABX probes, mpixel base (above) and mpixel large (below). Languages seen and unseen during pre-training, in different combinations.

Figure 9: ABX probes, mpixel base (above) and mpixel large (below), mpixel base pre-training languages. Pairs of languages that share script (blue) versus pairs of languages that have different scripts (orange).

Figure 10: Layer-wise accuracy of linear script-identification probes trained on the representations of each base model. Left (iid): unseen sentences from the 16 training languages. Right (ood): 32 unseen languages whose scripts were seen during training.

Figure 11: linspector probing results for the Case task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 12: linspector probing results for the Gender task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 13: linspector probing results for the Mood task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 14: linspector probing results for the Number task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 15: linspector probing results for the OddFeat task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 16: linspector probing results for the POS task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 17: linspector probing results for the Person task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 18: linspector probing results for the Possession task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 19: linspector probing results for the Pseudo task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 20: linspector probing results for the SameFeat task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 21: linspector probing results for the TagCount task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 22: linspector probing results for the Tense task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 23: linspector probing results for the Voice task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 24: linspector probing results for the Comparison task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Figure 25: linspector probing results for the Polarity task. Each panel shows linear probe accuracy per layer, with the x-axis giving the layer position relative to the model depth.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Aug 31, 2026Ran Zhang, Miryam de Lhoneux, Wessel PoelmanContinued Pretraining Low-Resource Language Processing
LAGOM·NLP, Department of Computer Science, KU Leuven
MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
Apr 13, 2026Chen Hu, Yintao Tai, Antonio Vergari +2Multilingual Language Models Autoregressive Language Modeling
School of Informatics, University of Edinburgh
Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models
Oct 16, 2026Haowen Tang, Wei Xia, Raoyuan Zhao +2Language Model Robustness LLM Fine-Tuning
Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)

Aug 31, 2026J/K move · Enter open · S save

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

Ran Zhang, Miryam de Lhoneux, Wessel Poelman

LAGOM·NLP, Department of Computer Science, KU Leuven

Continued Pretraining Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.

Abstract

Multilingual Language Models

Explore similar work

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models