Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models

Haowen Tang; Wei Xia; Raoyuan Zhao; Hinrich Schütze; Yihong Liu

cs.CLOct 16, 2026

Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models

Authors: Haowen Tang, Wei Xia, Raoyuan Zhao, Hinrich Schütze, Yihong Liu

Organizations: Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)

Abstract

Pixel-based language models process rendered text without relying on a fixed subword vocabulary, offering a promising approach to noisy and multilingual inputs. However, their practical value depends on whether they can be reliably adapted to downstream tasks. To this end, we systematically examine this question through fine-tuning behavior, typographical robustness, and zero-shot crosslingual transfer. Under our tested settings, pixel-based models require longer training and broader labelled-data coverage than text-based models, while delayed learning and learning-rate-dependent failures make adaptation less reliable. Our analyses further suggest that task-relevant information is less readily accessible in pretrained pixel representations, placing greater demands on fine-tuning. Once successfully adapted, however, pixel-based models lose substantially less accuracy under typographical noise and achieve higher target-language accuracy than English-pretrained text-based counterparts on two crosslingual benchmarks, despite weaker clean English performance. Together, these findings reveal a central trade-off: replacing tokenization with rendering reduces sensitivity to corrupted and unfamiliar inputs, but shifts a central bottleneck to downstream adaptation.

Figures & tables

Figure 1: Overview of the practical challenges and benefits of pixel-based language models. Top : compared with BERT, PIXEL is more sensitive to learning rate, learns more slowly, benefits more from additional labelled examples, and requires greater adaptation of its pretrained representations. Bottom : once successfully fine-tuned, PIXEL is more robust to typographical noise and achieves better crosslingual transfer performance.

Figure 2: Learning-rate sensitivity and fine-tuning duration. ( a ) Successful MNLI runs at epoch 10, out of ten seeds per learning rate. ( b ) Held-out accuracy of development-selected checkpoints from the same runs. Dots show individual runs; diamonds show means across all ten seeds. ( c ) Development accuracy at each epoch in one recorded MNLI run per model. PIXEL is less stable across learning rates and can require longer fine-tuning than BERT.

Figure 3: Development accuracy with increasing numbers of distinct labelled examples under a fixed budget of 15,000 updates. Lines show means and error bars show standard deviations across three seeds. Individual scores appear in Table 9 . Across both tasks, BERT performs strongly with fewer distinct examples, whereas PIXEL benefits more from broader data coverage.

Figure 4: Pretraining and downstream adaptation. ( a ) SST-2 development accuracy when training from random initialization ( Random ), training a classifier on a frozen pretrained encoder ( Frozen ), or fine-tuning the full pretrained model ( Full FT ). ( b ) MNLI training success across five repeated runs for two seeds, comparing the original PIXEL checkpoint with checkpoints previously adapted on PAWS-X or Taxi1500. ( c ) Mean MNLI learning curves; original runs are split by outcome. PIXEL benefits substantially from adapting its pretrained representations, while prior supervised downstream task training makes subsequent fine-tuning faster and more reliable.

Figure 5: Effects of typographical noise on input length and task performance. ( a ) Percentage change in mean input length relative to clean inputs, measured in subword tokens for BERT and image patches for PIXEL. ( b ) Accuracy loss as a percentage of each model’s clean accuracy. Colors distinguish BERT-base and PIXEL-base; solid lines with circles show SST-2, and dashed lines with triangles show MNLI. Complete model scores appear in Table 13 .

	XNLI							PAWS-X
Model	EN	DE	FR	ES	RU	ZH	Avg.	EN	DE	FR	ES	ZH	JA	KO	Avg.
BERT-base	76.8	38.9	38.7	40.5	34.1	35.8	37.6	93.6	75.3	79.0	76.8	52.9	57.1	57.4	66.4
PIXEL-base	65.7	43.2	50.2	49.1	41.4	33.5	43.5	89.1	77.2	80.8	80.5	64.7	61.6	61.9	71.1
RoBERTa	83.2	51.0	59.8	61.9	39.0	41.2	50.6	94.9	83.4	83.9	84.7	66.8	63.1	61.6	73.9
PIXEL-bigrams	60.6	44.9	47.4	48.6	42.4	34.1	43.5	89.4	77.9	79.8	80.9	57.1	56.8	61.6	69.0
mBERT	74.2	69.1	68.7	68.3	65.2	66.2	67.5	94.0	86.1	87.2	86.6	76.8	74.5	71.0	80.4

Table 1: Zero-shot crosslingual transfer performance (%). Avg. is the average over target languages. Bold compares BERT-base and PIXEL-base within each column; other models provide reference results. Despite lower English accuracy, PIXEL-base achieves higher target-language averages than BERT-base on both tasks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Version
Python	3.10.20
PyTorch	2.2.2+cu121
CUDA runtime (PyTorch)	12.1
Transformers	4.17.0
Datasets	2.14.6
NumPy	1.23.5

Appendix

Table 2: Software versions in the audited experimental environment. CUDA denotes the runtime used by PyTorch.

Model	Parameters (millions)
BERT-base	109.5
mBERT	177.9
RoBERTa-base	124.6
XLM-R-base	278.0
PIXEL-base	86.6
PIXEL-M4	86.6

Appendix

Table 3: Reference sizes of the classification models, rounded to 0.1 million parameters. All eight encoders have 12 transformer layers.

Analysis	Samples	Max. length	Batch $\times$ accum.	Peak LR	Warm-up	Budget / horizon
MNLI LR	24k	256	$32\times 4$	1e-6–5e-5, six points	100	1,880 / 15,040 updates
SST-2 coverage	2k/8k/24k	64	$32\times 4$	BERT 2e-5; PIXEL 3e-5	400	15,000 / 15,000 updates
MNLI coverage	8k/24k	256	$32\times 4$	Both 2e-5	400	15,000 / 15,000 updates

Appendix

Table 4: Controlled training settings. Maximum length counts tokens for BERT and rendered patch positions for PIXEL. Budget is the observed optimizer-update count; horizon determines linear decay. All rows use no early stopping, weight decay 0.01, and global gradient clipping at 1.0. Pretraining diagnostics use the separate recipes in Appendix A.6 .

Seed	Loss	Train	First	Held	Loss	Train	First	Held	Loss	Train	First	Held
	1e-6				5e-6				1e-5
13	1.071	41.0	–	35.25	0.815	63.9	6	60.10	0.591	75.8	4	66.55
21	1.090	37.6	–	32.85	0.922	57.0	9	55.80	1.099	33.5	–	35.45
42	1.074	40.8	–	37.30	0.829	63.1	7	61.05	0.571	76.5	5	67.70
87	1.080	40.1	–	34.20	1.097	34.5	–	34.55	0.727	68.6	7	64.40
100	1.082	39.6	–	32.45	1.098	34.3	–	33.10	0.631	73.7	5	64.40

Appendix

Table 5: Paired-seed MNLI results (lower tested LRs). Loss and online training accuracy (Train, %) are epoch-10 values. First is the first epoch meeting loss

<1

and accuracy

>50\%

; – denotes no crossing and

\dagger

a crossing followed by endpoint non-fitting. Held (%) evaluates the best-selection-dev checkpoint on the separate matched holdout, which can precede a collapse. Original failures are retained.

Seed	Loss	Train	First	Held	Loss	Train	First	Held	Loss	Train	First	Held
	3e-5				4e-5				5e-5
13	0.345	86.7	6	66.15	0.147	94.7	4	65.35	1.099	34.3	–	35.45
21	0.766	66.7	8	62.25	0.396	84.5	6	64.85	1.098	34.2	–	35.45
42	0.216	91.9	5	67.35	0.601	75.0	7	62.55	1.099	33.8	–	35.10
87	0.286	89.2	5	65.05	0.321	88.0	6	65.35	1.084	37.3	–	37.20
100	1.099	34.2	–	35.60	1.099	33.5	–	35.50	1.081	38.0	–	37.55

Appendix

Table 6: Paired-seed MNLI results (upper tested LRs). Loss and online training accuracy (Train, %) are epoch-10 values. First is the first epoch meeting loss

<1

and accuracy

>50\%

; – denotes no crossing and

\dagger

a crossing followed by endpoint non-fitting. Held (%) evaluates the best-selection-dev checkpoint on the separate matched holdout, which can precede a collapse. Original failures are retained.

	1e-6		5e-6		1e-5		3e-5		4e-5		5e-5
Seed	First	Held	First	Held	First	Held	First	Held	First	Held	First	Held
13	4	66.70	2	74.20	2	74.20	1	75.85	1	76.65	1	75.40
21	5	68.15	2	74.85	2	76.75	1	77.10	1	76.50	1	77.10
42	4	68.70	2	74.65	2	75.15	1	77.70	1	78.50	1	77.75
87	4	65.65	2	75.95	2	75.75	1	76.90	1	76.75	1	77.65
100	4	67.50	2	74.70	2	76.15	1	77.30	1	76.55	1	76.25

Appendix

Table 7: Matched BERT-base MNLI LR control: the same ten seeds, 24k examples, optimizer settings, 1,880 observed updates, and 15,040-update scheduler horizon as the PIXEL screen. First is the first epoch meeting online loss

<1

and accuracy

>50\%

; Held is matched holdout accuracy (%) of the best-selection-dev checkpoint. All 60 runs fit at epoch 10. SD denotes sample standard deviation in percentage points.

LR / warm-up	Fit	Matched	Mismatched
1e-5 / 100	8/10	59.63	58.91
1e-5 / 400	6/10	53.10	52.78
3e-5 / 100	8/10	61.55	61.88
3e-5 / 400	8/10	62.41	62.42

Appendix

Table 8: Warm-up follow-up: endpoint fits and all-ten mean best-checkpoint holdout accuracies (%). The 100-update cells are the original runs, and the 400-update cells are new runs. Selected baseline reruns are reported separately in the text.

Task	Model	Samples	Seed 42	Seed 1234	Seed 5678	Mean $\pm$ SD	Final train
MNLI	BERT	8000	73.45	73.55	73.25	$73.42\pm 0.15$	100.00–100.00
MNLI	BERT	24000	76.75	77.45	77.50	$77.23\pm 0.42$	99.97–99.98
MNLI	PIXEL-base	8000	60.85	57.65	59.55	$59.35\pm 1.61$	100.00–100.00
MNLI	PIXEL-base	24000	63.85	65.85	67.05	$65.58\pm 1.62$	99.71–99.98
SST-2	BERT	2000	90.37	89.11	90.14	$89.87\pm 0.67$	100.00–100.00
SST-2	BERT	8000	92.20	92.32	91.06	$91.86\pm 0.70$	100.00–100.00

Appendix

Table 9: Fixed-15,000-update coverage results (%). The three seed columns and mean

\pm

sample SD report best recorded development accuracy (matched split for MNLI). Final train gives the range of online training accuracies across the three seeds at update 15,000, not exact full-train re-evaluations. All seeds, including the weak SST-2 24k PIXEL seed, are retained.

Seed	Train	Dev	Test
42	99.98	85.67	85.12
1234	99.89	68.58	65.35
5678	99.98	86.81	85.83
13	99.95	86.58	–
21	99.99	87.73	–
100	99.97	86.24	–

Appendix

Table 10: PIXEL-base SST-2 follow-up with a fixed 24k training subset and 15,000 updates. All accuracies are percentages evaluated at the best-development checkpoint. Test uses all 1,821 sentences; dashes indicate unevaluated checkpoints.

Seed	Initialization	Fits	First fit	Train	Dev-m
100	Original	1/5	6	80.92	64.70
	PAWS-X	5/5	2	96.51	65.27
	Taxi1500	5/5	2	96.23	64.62
5678	Original	0/5	–	–	–
	PAWS-X	5/5	1	96.55	63.98
	Taxi1500	4/5	2	95.88	64.63

Appendix

Table 11: MNLI initialization intervention across five repeats per seed and initialization. Fits counts runs with training loss

<1

and accuracy

>50\%

at epoch 10. First fit is the epoch at which successful runs first meet this criterion. Train and Dev-m are mean epoch-10 training and matched-development accuracies (%) among successful runs only; the Original means use one run for seed 100, and seed 5678 has no successful Original run. Taxi1500 means for seed 5678 use its four successful runs. Failed Original runs remain near 33–35% training accuracy.

Seed	Epoch	Dev (%)	Epoch	Dev (%)
	BERT		PIXEL-base
13	3	75.60	10	64.75
21	6	75.80	$9^{\dagger}$	35.45
42	2	76.20	$3^{\dagger}$	35.50
87	9	76.90	23	66.25
100	3	76.05	21	65.20

Appendix

Table 12: Best-checkpoint epochs and matched-development accuracies in the longer-budget MNLI comparison.

\dagger

marks near-chance PIXEL runs. Medians use the seven seeds with PIXEL development accuracy above 45%, paired with the same BERT seeds.

Model	EN	DE	FR	ES	RU	ZH	KO
BERT	90.0	35.0	58.4	25.1	6.8	7.1	3.6
mBERT	88.4	92.2	87.2	86.7	82.5	89.2	81.9
PIXEL-base	83.5	28.7	48.4	50.2	17.6	9.1	7.1
PIXEL-bigrams	64.5	15.9	25.5	23.1	11.8	7.3	11.1
PIXEL-M4	79.0	30.4	52.7	32.5	40.3	27.8	17.3

Appendix

Table 14: SIB-200 clean zero-shot macro F1 (%). English-only fine-tuning uses all 701 training examples.

Model	EN	DE	FR	RU	ZH
BERT	73.9	24.3	31.5	23.4	23.4
mBERT	73.9	49.5	61.3	56.8	57.7
XLM-R	81.1	69.4	69.4	71.2	67.6
PIXEL-base	24.3	25.2	27.0	22.5	24.3
PIXEL-M4	60.4	18.9	26.1	28.8	29.7

Appendix

Table 15: Taxi1500 clean zero-shot accuracy (%). The English source task has 860 training examples; several PIXEL checkpoints have weak English accuracy.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 493 papers

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Aug 31, 2026Ran Zhang, Miryam de Lhoneux, Wessel PoelmanContinued Pretraining Low-Resource Language Processing
LAGOM·NLP, Department of Computer Science, KU Leuven
MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
Apr 13, 2026Chen Hu, Yintao Tai, Antonio Vergari +2Multilingual Language Models Autoregressive Language Modeling
School of Informatics, University of Edinburgh
Scaling Multilingual Pretraining of Pixel Language Models
Oct 21, 2026Ilker Kesen, Jenny Kunz, Kasra Malihi +1Multilingual Language Models Language Model Scaling
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen

Aug 31, 2026J/K move · Enter open · S save

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

Ran Zhang, Miryam de Lhoneux, Wessel Poelman

LAGOM·NLP, Department of Computer Science, KU Leuven

Continued Pretraining Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.

Abstract

Language Model Robustness

Explore similar work

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

Scaling Multilingual Pretraining of Pixel Language Models