Pixel-based language models process rendered text without relying on a fixed subword vocabulary, offering a promising approach to noisy and multilingual inputs. However, their practical value depends on whether they can be reliably adapted to downstream tasks. To this end, we systematically examine this question through fine-tuning behavior, typographical robustness, and zero-shot crosslingual transfer. Under our tested settings, pixel-based models require longer training and broader labelled-data coverage than text-based models, while delayed learning and learning-rate-dependent failures make adaptation less reliable. Our analyses further suggest that task-relevant information is less readily accessible in pretrained pixel representations, placing greater demands on fine-tuning. Once successfully adapted, however, pixel-based models lose substantially less accuracy under typographical noise and achieve higher target-language accuracy than English-pretrained text-based counterparts on two crosslingual benchmarks, despite weaker clean English performance. Together, these findings reveal a central trade-off: replacing tokenization with rendering reduces sensitivity to corrupted and unfamiliar inputs, but shifts a central bottleneck to downstream adaptation.
Figures & tables
Figure 1: Overview of the practical challenges and benefits of pixel-based language models. Top : compared with BERT, PIXEL is more sensitive to learning rate, learns more slowly, benefits more from additional labelled examples, and requires greater adaptation of its pretrained representations. Bottom : once successfully fine-tuned, PIXEL is more robust to typographical noise and achieves better crosslingual transfer performance.
Figure 2: Learning-rate sensitivity and fine-tuning duration. ( a ) Successful MNLI runs at epoch 10, out of ten seeds per learning rate. ( b ) Held-out accuracy of development-selected checkpoints from the same runs. Dots show individual runs; diamonds show means across all ten seeds. ( c ) Development accuracy at each epoch in one recorded MNLI run per model. PIXEL is less stable across learning rates and can require longer fine-tuning than BERT.
Figure 3: Development accuracy with increasing numbers of distinct labelled examples under a fixed budget of 15,000 updates. Lines show means and error bars show standard deviations across three seeds. Individual scores appear in Table 9 . Across both tasks, BERT performs strongly with fewer distinct examples, whereas PIXEL benefits more from broader data coverage.
Figure 4: Pretraining and downstream adaptation. ( a ) SST-2 development accuracy when training from random initialization ( Random ), training a classifier on a frozen pretrained encoder ( Frozen ), or fine-tuning the full pretrained model ( Full FT ). ( b ) MNLI training success across five repeated runs for two seeds, comparing the original PIXEL checkpoint with checkpoints previously adapted on PAWS-X or Taxi1500. ( c ) Mean MNLI learning curves; original runs are split by outcome. PIXEL benefits substantially from adapting its pretrained representations, while prior supervised downstream task training makes subsequent fine-tuning faster and more reliable.
Figure 5: Effects of typographical noise on input length and task performance. ( a ) Percentage change in mean input length relative to clean inputs, measured in subword tokens for BERT and image patches for PIXEL. ( b ) Accuracy loss as a percentage of each model’s clean accuracy. Colors distinguish BERT-base and PIXEL-base; solid lines with circles show SST-2, and dashed lines with triangles show MNLI. Complete model scores appear in Table 13 .
XNLI
PAWS-X
Model
EN
DE
FR
ES
RU
ZH
Avg.
EN
DE
FR
ES
ZH
JA
KO
Avg.
BERT-base
76.8
38.9
38.7
40.5
34.1
35.8
37.6
93.6
75.3
79.0
76.8
52.9
57.1
57.4
66.4
PIXEL-base
65.7
43.2
50.2
49.1
41.4
33.5
43.5
89.1
77.2
80.8
80.5
64.7
61.6
61.9
71.1
RoBERTa
83.2
51.0
59.8
61.9
39.0
41.2
50.6
94.9
83.4
83.9
84.7
66.8
63.1
61.6
73.9
PIXEL-bigrams
60.6
44.9
47.4
48.6
42.4
34.1
43.5
89.4
77.9
79.8
80.9
57.1
56.8
61.6
69.0
mBERT
74.2
69.1
68.7
68.3
65.2
66.2
67.5
94.0
86.1
87.2
86.6
76.8
74.5
71.0
80.4
Table 1: Zero-shot crosslingual transfer performance (%). Avg. is the average over target languages. Bold compares BERT-base and PIXEL-base within each column; other models provide reference results. Despite lower English accuracy, PIXEL-base achieves higher target-language averages than BERT-base on both tasks.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Version
Python
3.10.20
PyTorch
2.2.2+cu121
CUDA runtime (PyTorch)
12.1
Transformers
4.17.0
Datasets
2.14.6
NumPy
1.23.5
Appendix
Table 2: Software versions in the audited experimental environment. CUDA denotes the runtime used by PyTorch.
Model
Parameters (millions)
BERT-base
109.5
mBERT
177.9
RoBERTa-base
124.6
XLM-R-base
278.0
PIXEL-base
86.6
PIXEL-M4
86.6
Appendix
Table 3: Reference sizes of the classification models, rounded to 0.1 million parameters. All eight encoders have 12 transformer layers.
Analysis
Samples
Max. length
Batch × accum.
Peak LR
Warm-up
Budget / horizon
MNLI LR
24k
256
32×4
1e-6–5e-5, six points
100
1,880 / 15,040 updates
SST-2 coverage
2k/8k/24k
64
32×4
BERT 2e-5; PIXEL 3e-5
400
15,000 / 15,000 updates
MNLI coverage
8k/24k
256
32×4
Both 2e-5
400
15,000 / 15,000 updates
Appendix
Table 4: Controlled training settings. Maximum length counts tokens for BERT and rendered patch positions for PIXEL. Budget is the observed optimizer-update count; horizon determines linear decay. All rows use no early stopping, weight decay 0.01, and global gradient clipping at 1.0. Pretraining diagnostics use the separate recipes in Appendix A.6 .
1e-6
5e-6
1e-5
Seed
Loss
Train
First
Held
Loss
Train
First
Held
Loss
Train
First
Held
13
1.071
41.0
–
35.25
0.815
63.9
6
60.10
0.591
75.8
4
66.55
21
1.090
37.6
–
32.85
0.922
57.0
9
55.80
1.099
33.5
–
35.45
42
1.074
40.8
–
37.30
0.829
63.1
7
61.05
0.571
76.5
5
67.70
87
1.080
40.1
–
34.20
1.097
34.5
–
34.55
0.727
68.6
7
64.40
100
1.082
39.6
–
32.45
1.098
34.3
–
33.10
0.631
73.7
5
64.40
Appendix
Table 5: Paired-seed MNLI results (lower tested LRs). Loss and online training accuracy (Train, %) are epoch-10 values. First is the first epoch meeting loss <1 and accuracy >50% ; – denotes no crossing and † a crossing followed by endpoint non-fitting. Held (%) evaluates the best-selection-dev checkpoint on the separate matched holdout, which can precede a collapse. Original failures are retained.
3e-5
4e-5
5e-5
Seed
Loss
Train
First
Held
Loss
Train
First
Held
Loss
Train
First
Held
13
0.345
86.7
6
66.15
0.147
94.7
4
65.35
1.099
34.3
–
35.45
21
0.766
66.7
8
62.25
0.396
84.5
6
64.85
1.098
34.2
–
35.45
42
0.216
91.9
5
67.35
0.601
75.0
7
62.55
1.099
33.8
–
35.10
87
0.286
89.2
5
65.05
0.321
88.0
6
65.35
1.084
37.3
–
37.20
100
1.099
34.2
–
35.60
1.099
33.5
–
35.50
1.081
38.0
–
37.55
Appendix
Table 6: Paired-seed MNLI results (upper tested LRs). Loss and online training accuracy (Train, %) are epoch-10 values. First is the first epoch meeting loss <1 and accuracy >50% ; – denotes no crossing and † a crossing followed by endpoint non-fitting. Held (%) evaluates the best-selection-dev checkpoint on the separate matched holdout, which can precede a collapse. Original failures are retained.
1e-6
5e-6
1e-5
3e-5
4e-5
5e-5
Seed
First
Held
First
Held
First
Held
First
Held
First
Held
First
Held
13
4
66.70
2
74.20
2
74.20
1
75.85
1
76.65
1
75.40
21
5
68.15
2
74.85
2
76.75
1
77.10
1
76.50
1
77.10
42
4
68.70
2
74.65
2
75.15
1
77.70
1
78.50
1
77.75
87
4
65.65
2
75.95
2
75.75
1
76.90
1
76.75
1
77.65
100
4
67.50
2
74.70
2
76.15
1
77.30
1
76.55
1
76.25
Appendix
Table 7: Matched BERT-base MNLI LR control: the same ten seeds, 24k examples, optimizer settings, 1,880 observed updates, and 15,040-update scheduler horizon as the PIXEL screen. First is the first epoch meeting online loss <1 and accuracy >50% ; Held is matched holdout accuracy (%) of the best-selection-dev checkpoint. All 60 runs fit at epoch 10. SD denotes sample standard deviation in percentage points.
LR / warm-up
Fit
Matched
Mismatched
1e-5 / 100
8/10
59.63
58.91
1e-5 / 400
6/10
53.10
52.78
3e-5 / 100
8/10
61.55
61.88
3e-5 / 400
8/10
62.41
62.42
Appendix
Table 8: Warm-up follow-up: endpoint fits and all-ten mean best-checkpoint holdout accuracies (%). The 100-update cells are the original runs, and the 400-update cells are new runs. Selected baseline reruns are reported separately in the text.
Task
Model
Samples
Seed 42
Seed 1234
Seed 5678
Mean ± SD
Final train
MNLI
BERT
8000
73.45
73.55
73.25
73.42±0.15
100.00–100.00
MNLI
BERT
24000
76.75
77.45
77.50
77.23±0.42
99.97–99.98
MNLI
PIXEL-base
8000
60.85
57.65
59.55
59.35±1.61
100.00–100.00
MNLI
PIXEL-base
24000
63.85
65.85
67.05
65.58±1.62
99.71–99.98
SST-2
BERT
2000
90.37
89.11
90.14
89.87±0.67
100.00–100.00
SST-2
BERT
8000
92.20
92.32
91.06
91.86±0.70
100.00–100.00
Appendix
Table 9: Fixed-15,000-update coverage results (%). The three seed columns and mean ± sample SD report best recorded development accuracy (matched split for MNLI). Final train gives the range of online training accuracies across the three seeds at update 15,000, not exact full-train re-evaluations. All seeds, including the weak SST-2 24k PIXEL seed, are retained.
Seed
Train
Dev
Test
42
99.98
85.67
85.12
1234
99.89
68.58
65.35
5678
99.98
86.81
85.83
13
99.95
86.58
–
21
99.99
87.73
–
100
99.97
86.24
–
Appendix
Table 10: PIXEL-base SST-2 follow-up with a fixed 24k training subset and 15,000 updates. All accuracies are percentages evaluated at the best-development checkpoint. Test uses all 1,821 sentences; dashes indicate unevaluated checkpoints.
Seed
Initialization
Fits
First fit
Train
Dev-m
100
Original
1/5
6
80.92
64.70
PAWS-X
5/5
2
96.51
65.27
Taxi1500
5/5
2
96.23
64.62
5678
Original
0/5
–
–
–
PAWS-X
5/5
1
96.55
63.98
Taxi1500
4/5
2
95.88
64.63
Appendix
Table 11: MNLI initialization intervention across five repeats per seed and initialization. Fits counts runs with training loss <1 and accuracy >50% at epoch 10. First fit is the epoch at which successful runs first meet this criterion. Train and Dev-m are mean epoch-10 training and matched-development accuracies (%) among successful runs only; the Original means use one run for seed 100, and seed 5678 has no successful Original run. Taxi1500 means for seed 5678 use its four successful runs. Failed Original runs remain near 33–35% training accuracy.
BERT
PIXEL-base
Seed
Epoch
Dev (%)
Epoch
Dev (%)
13
3
75.60
10
64.75
21
6
75.80
9†
35.45
42
2
76.20
3†
35.50
87
9
76.90
23
66.25
100
3
76.05
21
65.20
Appendix
Table 12: Best-checkpoint epochs and matched-development accuracies in the longer-budget MNLI comparison. † marks near-chance PIXEL runs. Medians use the seven seeds with PIXEL development accuracy above 45%, paired with the same BERT seeds.
Model
EN
DE
FR
ES
RU
ZH
KO
BERT
90.0
35.0
58.4
25.1
6.8
7.1
3.6
mBERT
88.4
92.2
87.2
86.7
82.5
89.2
81.9
PIXEL-base
83.5
28.7
48.4
50.2
17.6
9.1
7.1
PIXEL-bigrams
64.5
15.9
25.5
23.1
11.8
7.3
11.1
PIXEL-M4
79.0
30.4
52.7
32.5
40.3
27.8
17.3
Appendix
Table 14: SIB-200 clean zero-shot macro F1 (%). English-only fine-tuning uses all 701 training examples.
Model
EN
DE
FR
RU
ZH
BERT
73.9
24.3
31.5
23.4
23.4
mBERT
73.9
49.5
61.3
56.8
57.7
XLM-R
81.1
69.4
69.4
71.2
67.6
PIXEL-base
24.3
25.2
27.0
22.5
24.3
PIXEL-M4
60.4
18.9
26.1
28.8
29.7
Appendix
Table 15: Taxi1500 clean zero-shot accuracy (%). The English source task has 860 training examples; several PIXEL checkpoints have weak English accuracy.
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.
Ran Zhang, Miryam de Lhoneux, Wessel Poelman
LAGOM·NLP, Department of Computer Science, KU Leuven
Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at https://github.com/april-tools/mixar.
Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.
Ilker Kesen, Jenny Kunz, Kasra Malihi +1
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen