Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at https://github.com/april-tools/mixar.
Figures & tables
Figure 1: Mixar : a Transformer-based decoder-only architecture that uses rendered text as input to learn across multiple languages. Encoding language as pixels, enables Mixar to be robust to visual attacks as well.
Figure 2: Increasing the patch-size in Mixar helps with non-English scripts as shown for patches of size 8 × 8 pixels (left) which cannot capture fine-grained details for Chinese (top), Korean (middle) and Japanese (bottom) characters, while 32 × 32 pixel patches can.
Model
Params
Patch Size
Seen
Unseen
CJK
Latin
AVG
Latin
zh
ja
ko
AVG
de
en
es
it
fr
AVG
fi
tr
uz
MGPT-2
124M
NA
65.9
81.1
61.1
69.4
81.7
87.7
81.7
81.6
81.2
82.8
77.8
44.9
45.2
40.5
GPT-2
124M
NA
42.2
44.6
22.3
36.4
39.4
79.4
38.4
28.6
50.2
47.2
43.1
17.3
20.3
33.3
Pixar stage2
113M
8 × 8
11.0
25.9
16.6
17.8
40.6
81.5
52.3
53.8
60.5
57.7
42.8
29.2
39.7
42.9
Mixar stage1
113M
8 × 8
58.0
75.3
38.8
57.4
78.1
77.9
79.3
75.3
83.2
78.8
70.7
54.4
53.1
58.2
Table 1: On SIB-200, Mixar clearly outperforms all baselines on unseen languages, and most of all on low-resource ones, including the tokenizer-based MGPT-2 pretrained on the very same corpus. We report the performance on both seen and unseen languages from SIB-200 , categorized by script families. “AVG” denotes the average score within the respective language group, “Non-Eng AVG” represents the average performance excluding English, and “All AVG” is the mean over all 22 evaluated languages (8 seen and 14 unseen). Languages marked † are low-resource (class ≤ 2 in the taxonomy of Joshi et al., 2020 ), and “Low-res AVG” is their mean; Table 7 maps language codes to languages.
Models
Parameters
Patch size
seen
unseen (Translate-Train)
Non-Eng AVG
de
en
es
fr
zh
AVG
bg
el
ur
tr
ru
AVG
Pixel-M4
112M
16 × 16
NA
NA
NA
NA
NA
NA
52.1
52.9
47.3
52.2
38.4
48.6
NA
MGPT-2
124M
NA
72.7
78.2
73.0
71.4
67.1
72.5
66.9
67.0
56.1
65.7
63.8
63.9
67.1
BiLSTM-max
NA
NA
66.5
73.7
68.8
68.3
67.0
NA
67.4
66.4
56.6
64.5
66.5
64.3
65.8
multi-BERT
110M
NA
75.9
81.9
77.8
NA
76.7
NA
NA
NA
61.6
NA
NA
NA
NA
Pixar stage2
85M
8 × 8
67.2
78.8
69.8
67.7
57.3
NA
60.5
64.6
50.2
65.5
64.0
61.0
63.0
Table 2: 477M Mixar outperforms the MGPT-2 baseline on seen and unseen languages on XNLI as shown by accuracy values, especially the average non-English ones.
Models
Parameters
Patch size
MNLI-m/mm
QQP
QNLI
SST-2
COLA
STSB
MRPC
RTE
WNLI
AVG
(pixel)
392k
363k
108k
67k
8.5k
5.7k
3.5k
2.5k
635
GPT-2
126M
NA
81.0
89.4
87.7
92.5
77.0
74.9
71.5
52.0
54.9
75.6
MGPT-2
126M
NA
78.2/78.5
85.2
85.3
87.9
33.7
87.0
82.3
64.6
57.4
74.0
BERT
110M
NA
84.0/84.2
87.6
91.0
92.6
60.3
88.8
90.2
69.5
51.8
80.0
Pixel
86M
16 × 16
78.1/78.9
84.5
87.8
89.6
38.4
81.1
88.2
60.5
53.8
74.1
Pixar stage1
85M
8 × 8
78.4/78.6
85.6
85.7
89.0
39.9
81.7
83.3
58.5
59.2
74.0
Table 3: Mixar performs better than Pixar and MGPT-2 on the GLUE benchmark as shown by Matthew’s correlation for COLA, Spearman’s ρ for STSB and F1 score values for MRPC and QQP, while other tasks used accuracy. The number below the each tasks denotes the amount of training samples per task.
en
de
es
fr
it
zh
ja
ko
characters per patch
1.59
1.58
1.59
1.60
1.66
1.02
1.01
1.24
GPT-2 tokens per patch
0.42
0.62
0.58
0.56
0.58
1.77
1.27
2.29
Table 4: A CJK patch carries three to five times more text than an English one. Characters and GPT-2 tokens contained in a single rendered 32×32 patch, measured on mC4 samples of the eight pretraining languages. MGPT-2 shares the GPT-2 tokenizer, so its token counts are identical.
Models
#Params
Patch
bAbI
LAMBADA
size
en
zh
ja
ko
en
de
es
fr
it
avg
GPT-2
124M
-
26.8
17.4
17.4
25.6
17.1
4.3
6.1
7.9
6.3
8.3
MGPT-2
124M
-
24.3
17.6
21.8
10.6
18.8
18.3
16.0
20.6
17.3
18.2
Pixar stage2
113M
8 × 8
19.6
0.0
0.0
0.0
13.8
3.6
2.5
5.9
5.6
6.3
Mixar stage1
113M
8 × 8
11.4
0.0
0.0
0.0
3.8
10.8
4.6
5.0
5.8
6.0
Mixar stage1
116M
32 × 32
15.2
3.8
2.0
2.2
2.4
8.6
3.8
4.2
4.6
4.7
Table 5: Scaling Mixar to 477M outperforms the English-only Pixar and text-based GPT-2 on average across multilingual generative tasks. For bAbI, we report few-shot free-generation accuracy (dash-delimited answers; 2 shots for the 477M model, 3 for the others) and for LAMBADA zero-shot last-word free-generation accuracy. Mixar generations are judged by a Qwen3.6-35B OCR judge with a pixel-exact shortcut and confusable folding, reading only the generated region on bAbI and the generation with a short anchoring context on LAMBADA, so that the judge never sees the question it is transcribing an answer to ( Appendix L ); the stage2-gan row is the adversarial stage-2 counterpart of the 8 × 8 model ( Appendices J and K for training details). Pixar numbers are from Tai et al. (2024) .
Figure 3: Mixar can handle multilingual text as image , as shown here for examples of correct completions (black) for German, Spanish and Italian prompts (gray) on LAMBADA.
Figure 4: Mixar withstands English orthographic attacks more gracefully than tokenizer-based models : accuracy under increasing attack ratios on LAMBADA and bAbI (free-generation protocol, lookalike glyphs from the bundled fonts). Attack examples are shown in Fig. 6 .
Figure 5: An orthographic attack barely changes the rendered image. The same English LAMBADA prompt rendered before (left) and after (right) an attack at ratio 1.0 , which replaces every attackable character by a Unicode lookalike. The text a tokenizer reads is almost entirely different, while the pixels Mixar reads are nearly unchanged.
Table 6: This table shows the letters outside the English alphabet for other 4 latin languages of pretraining dataset.
Code
Language
Script
Pretraining
de
German
Latin
seen
en
English
Latin
seen
es
Spanish
Latin
seen
fr
French
Latin
seen
it
Italian
Latin
seen
ja
Japanese
Kana + Han
seen
Appendix
Table 7: Language codes used in the evaluation tables, with script and pretraining status. † : low-resource languages (class ≤ 2, Joshi et al., 2020 ).
Unicode block
PixeloidSans
Go Noto Current
Basic Latin (ASCII)
95/128
98/128
Latin-1 Supplement
117/128
96/128
Latin Extended-A
128/128
128/128
Latin Extended-B
97/208
208/208
Greek and Coptic
111/144
135/144
Cyrillic (+ Supplement)
174/304
304/304
Appendix
Table 8: Per-block character coverage (covered/total codepoints) of the two main bundled fonts of our rendering tool. The primary PixeloidSans font covers Latin scripts; Go Noto Current supplies the remaining scripts, including full kana and Hangul coverage and the common CJK ideographs.
Render Configuration
Model Structure
Stage 1 Hyperparameters
patch length
2
layers
12 / 24
peak lr
3e-4
patch number
360
attention heads
12 / 14
min. lr
3e-5
render DPI
80
hidden size
768 / 896
lr scheduler
CosineAnnealing
font size
32
activation
SwiGLU
optimizer
AdamW
patch size
32
intermediate size
3072 / 6656
β1
0.9
font
PixeloidSans
parameters
116M / 477M
β2
0.95
Appendix
Table 9: Rendering configuration, model structure and stage-1 pretraining hyperparameters of the 116M / 477M Mixar models. Stage-2 (GRPO) hyperparameters are reported separately in Table 11 .
Stage-2 objective (113M, 8 × 8)
steps
TF exact
none (stage-1 baseline)
–
5.1
GAN, adaptive λ=0.1 , threshold alt.
800
5.7
GAN, adaptive λ=0.3 , 1:1 alt.
800
5.6
GAN, adaptive λ=0.3 , 1:1 alt.
5000
7.4
GAN, adaptive λ=0.3 , 1:1 alt.
50000
6.9
GAN, adaptive λ=0.3 , threshold alt.
800
5.6
Appendix
Table 10: Stage-2 objective ablation for the 8 × 8 Mixar : teacher-forced LAMBADA last-word exact-match (%, 2,500 passages, 5 languages). All GAN variants use the straight-through estimator through the discriminator and the adaptive gradient-ratio weight; without it every λm∈{1,3,9} diverges within 200 steps.
113M (8 × 8)
116M (32 × 32)
477M (32 × 32)
initialization
GRPO 50k ckpt
4-generation chain
1-generation chain
batch / micro-batch
128 / 32
32 / 8
128 / 8
learning rate
10−5
5×10−6
10−5
steps per generation
2000
1500–4000
2000
group size G
8
16
8
reward blend dblend
0.5
0.7
0.5
Appendix
Table 11: Per-model rollout-GRPO hyperparameters of the released stage-2 models. The word gap is the all-white column run that marks a word boundary, calibrated to each font size.
Figure 6: Examples of 5 latin languages contains in pretraining dataset. 20% letters in these letters are changed by confused one.
stage1
MNLI
QQP
QNLI
SST-2
COLA
STSB
MRPC
RTE
WNLI
85M Pixar stage1 lr
3e-5
3e-5
3e-5
3e-5
3e-5
3e-5
6e-5
3e-5
3e-5
116M Mixar stage1 lr
3e-5
3e-5
3e-5
3e-5
3e-5
3e-5
6e-5
3e-5
6e-5
477M Mixar stage1 lr
3e-5
3e-5
3e-5
3e-5
6e-5
3e-5
6e-5
3e-5
6e-5
Weight decay
0.1
0.1
0.1
0.01
0.01
0.01
0.01
0.01
0.01
Optimizer
AdamW
Warmup
Linear warmup
Appendix
Table 12: This table shows the hyperparameters applied in the GLUE benchmark.
Models
lr
Weight decay
Optimizer
Warmup
Warmup steps
β1
β2
Random seed
Batch size
Max steps
evaluation freq.
Mixar stage1
3e-5
0.1
AdamW
Linear warmup
1000
0.9
0.95
42
256
8000
500
Mixar stage2
3e-5
0.1
AdamW
Linear warmup
1000
0.9
0.95
42
256
8000
500
Appendix
Table 13: This table shows the hyperparameters used in XNLI downstream tasks.
Models
lr
Weight decay
Optimizer
Warmup
Warmup steps
β1
β2
Random seed
Batch size
Max steps
evaluation freq.
Mixar stage1
5e-5
0.01
AdamW
Linear warmup
50
0.9
0.95
42
128
500
25
Appendix
Table 14: This table shows the hyperparameters used in SIB downstream tasks.
attack
LAMBADA
bAbI
ratio
GPT-2
MGPT-2
Mixar (116M)
Mixar (477M)
GPT-2
MGPT-2
Mixar (116M)
Mixar (477M)
0.0
32.6
26.2
9.2
13.4
41.4
38.0
16.6
34.6
0.01
30.0
24.6
9.0
12.6
43.2
36.2
17.0
34.6
0.05
19.0
15.6
8.8
13.2
35.8
24.2
15.0
23.8
0.1
8.4
7.2
7.6
9.6
23.0
14.0
13.2
18.6
0.2
1.6
2.2
4.8
7.4
8.0
4.2
11.2
11.4
Appendix
Table 15: Accuracy under the orthographic attack for each attack ratio (English LAMBADA and bAbI, free-generation protocol, lookalike glyphs restricted to the bundled fonts). The Mixar entries are the stage-2 models; MGPT-2, which shares our pretraining corpus, receives the identical attacked text. Prior-work numbers (e.g., Pixar ) are not comparable under this protocol and are omitted.
Models
Parameters
Patch size
MNLI-m/mm
QQP
QNLI
SST-2
COLA
STSB
MRPC
RTE
WNLI
AVG
392k
363k
108k
67k
8.5k
5.7k
3.5k
2.5k
635
PIXAR stage1
85M
8 × 8
76.4 / 77.6
84.3
84.2
88.0
27.7
81.2
82.5
58.5
56.3
71.6
Mixar stage1
116M
32 × 32
72.5 / 72.6
83.2
82.9
83.6
17.6
80.6
83.1
61.0
56.3
69.3
Mixar stage1
474M
8 × 8
78.6 / 78.0
85.6
85.1
88.8
28.5
82.9
82.9
65.3
56.3
73.2
Mixar stage1
477M
32 × 32
76.1 / 76.1
85.3
84.9
87.7
23.2
82.8
83.2
63.9
56.3
72.0
Appendix
Table 16: This table shows the comparison of Pixar and Mixar with the patch size of 8 × 8 and 32 × 32 trained on 0.1M steps and finetuned on GLUE tasks. Matthew’s correlation is used for COLA and Spearman’s ρ is applied for STSB. The F1 score is for MRPC and QQP. Other tasks used accuracy.
Dataset
Number of patches
chars/patch
GPT2 tok/patch
Chinese mc4
3,345,800,998
2.47
4.40
English mc4
17,387,543,604
3.26
0.81
French mc4
23,633,038,801
3.28
1.16
German mc4
28,550,835,190
3.20
1.19
Italian mc4
12,161,721,091
3.35
1.20
Spanish mc4
33,542,982,712
3.25
1.16
Appendix
Table 17: Patch-level data statistics for the Mixar pretraining languages in the multilingual mC4 dataset, including average characters and GPT-2 tokens per patch.
Figure 7: This image shows the comparison of patch size 8 and 32 for all pretraining scripts.
Figure 8: This image shows the correct samples of English, Chinese and Japanese bAbI tasks
Figure 9: This image shows the wrong samples of English, Chinese and Japanese bAbI tasks
Figure 10: This image shows some correct samples of LAMBADA task of eight pretraining languages
Figure 11: This image shows some wrong samples of LAMBADA task of eight pretraining languages
Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.
Ilker Kesen, Jenny Kunz, Kasra Malihi +1
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen
Pixel-based language models process rendered text without relying on a fixed subword vocabulary, offering a promising approach to noisy and multilingual inputs. However, their practical value depends on whether they can be reliably adapted to downstream tasks. To this end, we systematically examine this question through fine-tuning behavior, typographical robustness, and zero-shot crosslingual transfer. Under our tested settings, pixel-based models require longer training and broader labelled-data coverage than text-based models, while delayed learning and learning-rate-dependent failures make adaptation less reliable. Our analyses further suggest that task-relevant information is less readily accessible in pretrained pixel representations, placing greater demands on fine-tuning. Once successfully adapted, however, pixel-based models lose substantially less accuracy under typographical noise and achieve higher target-language accuracy than English-pretrained text-based counterparts on two crosslingual benchmarks, despite weaker clean English performance. Together, these findings reveal a central trade-off: replacing tokenization with rendering reduces sensitivity to corrupted and unfamiliar inputs, but shifts a central bottleneck to downstream adaptation.
Haowen Tang, Wei Xia, Raoyuan Zhao +2
Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.
Ran Zhang, Miryam de Lhoneux, Wessel Poelman
LAGOM·NLP, Department of Computer Science, KU Leuven