MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

Chen Hu; Yintao Tai; Antonio Vergari; Frank Keller; Alessandro Suglia

cs.CLApr 13, 2026

MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

Authors: Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro Suglia

Organizations: School of Informatics, University of Edinburgh

Abstract

Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at https://github.com/april-tools/mixar.

Figures & tables

Figure 1: Mixar : a Transformer-based decoder-only architecture that uses rendered text as input to learn across multiple languages. Encoding language as pixels, enables Mixar to be robust to visual attacks as well.

Figure 2: Increasing the patch-size in Mixar helps with non-English scripts as shown for patches of size 8 $\times$ 8 pixels (left) which cannot capture fine-grained details for Chinese (top), Korean (middle) and Japanese (bottom) characters, while 32 $\times$ 32 pixel patches can.

Model	Params	Patch Size	Seen											Unseen
			CJK				Latin						AVG	Latin
			zh	ja	ko	AVG	de	en	es	it	fr	AVG	AVG	fi	tr	uz
MGPT-2	124M	NA	65.9	81.1	61.1	69.4	81.7	87.7	81.7	81.6	81.2	82.8	77.8	44.9	45.2	40.5
GPT-2	124M	NA	42.2	44.6	22.3	36.4	39.4	79.4	38.4	28.6	50.2	47.2	43.1	17.3	20.3	33.3
Pixar ${}_{\text{stage2}}$	113M	8 $\times$ 8	11.0	25.9	16.6	17.8	40.6	81.5	52.3	53.8	60.5	57.7	42.8	29.2	39.7	42.9
Mixar ${}_{\text{stage1}}$	113M	8 $\times$ 8	58.0	75.3	38.8	57.4	78.1	77.9	79.3	75.3	83.2	78.8	70.7	54.4	53.1	58.2

Table 1: On SIB-200, Mixar clearly outperforms all baselines on unseen languages, and most of all on low-resource ones, including the tokenizer-based MGPT-2 pretrained on the very same corpus. We report the performance on both seen and unseen languages from SIB-200 , categorized by script families. “AVG” denotes the average score within the respective language group, “Non-Eng AVG” represents the average performance excluding English, and “All AVG” is the mean over all 22 evaluated languages (8 seen and 14 unseen). Languages marked † are low-resource (class

\leq

2 in the taxonomy of Joshi et al., 2020 ), and “Low-res AVG” is their mean; Table 7 maps language codes to languages.

Models	Parameters	Patch size	seen						unseen (Translate-Train)						Non-Eng AVG
Models	Parameters	Patch size	de	en	es	fr	zh	AVG	bg	el	ur	tr	ru	AVG	Non-Eng AVG
Pixel-M4	112M	16 $\times$ 16	NA	NA	NA	NA	NA	NA	52.1	52.9	47.3	52.2	38.4	48.6	NA
MGPT-2	124M	NA	72.7	78.2	73.0	71.4	67.1	72.5	66.9	67.0	56.1	65.7	63.8	63.9	67.1
BiLSTM-max	NA	NA	66.5	73.7	68.8	68.3	67.0	NA	67.4	66.4	56.6	64.5	66.5	64.3	65.8
multi-BERT	110M	NA	75.9	81.9	77.8	NA	76.7	NA	NA	NA	61.6	NA	NA	NA	NA
Pixar ${}_{\text{stage2}}$	85M	8 $\times$ 8	67.2	78.8	69.8	67.7	57.3	NA	60.5	64.6	50.2	65.5	64.0	61.0	63.0

Table 2: 477M Mixar outperforms the MGPT-2 baseline on seen and unseen languages on XNLI as shown by accuracy values, especially the average non-English ones.

Models	Parameters	Patch size	MNLI-m/mm	QQP	QNLI	SST-2	COLA	STSB	MRPC	RTE	WNLI	AVG
Models	Parameters	(pixel)	392k	363k	108k	67k	8.5k	5.7k	3.5k	2.5k	635	AVG
GPT-2	126M	NA	81.0	89.4	87.7	92.5	77.0	74.9	71.5	52.0	54.9	75.6
MGPT-2	126M	NA	78.2/78.5	85.2	85.3	87.9	33.7	87.0	82.3	64.6	57.4	74.0
BERT	110M	NA	84.0/84.2	87.6	91.0	92.6	60.3	88.8	90.2	69.5	51.8	80.0
Pixel	86M	16 $\times$ 16	78.1/78.9	84.5	87.8	89.6	38.4	81.1	88.2	60.5	53.8	74.1
Pixar ${}_{\text{stage1}}$	85M	8 $\times$ 8	78.4/78.6	85.6	85.7	89.0	39.9	81.7	83.3	58.5	59.2	74.0

Table 3: Mixar performs better than Pixar and MGPT-2 on the GLUE benchmark as shown by Matthew’s correlation for COLA, Spearman’s

\rho

for STSB and F1 score values for MRPC and QQP, while other tasks used accuracy. The number below the each tasks denotes the amount of training samples per task.

	en	de	es	fr	it	zh	ja	ko
characters per patch	1.59	1.58	1.59	1.60	1.66	1.02	1.01	1.24
GPT-2 tokens per patch	0.42	0.62	0.58	0.56	0.58	1.77	1.27	2.29

Table 4: A CJK patch carries three to five times more text than an English one. Characters and GPT-2 tokens contained in a single rendered

32{\times}32

patch, measured on mC4 samples of the eight pretraining languages. MGPT-2 shares the GPT-2 tokenizer, so its token counts are identical.

Models	#Params	Patch	bAbI				LAMBADA
		size	en	zh	ja	ko	en	de	es	fr	it	avg
GPT-2	124M	-	26.8	17.4	17.4	25.6	17.1	4.3	6.1	7.9	6.3	8.3
MGPT-2	124M	-	24.3	17.6	21.8	10.6	18.8	18.3	16.0	20.6	17.3	18.2
Pixar ${}_{\text{stage2}}$	113M	8 $\times$ 8	19.6	0.0	0.0	0.0	13.8	3.6	2.5	5.9	5.6	6.3
Mixar ${}_{\text{stage1}}$	113M	8 $\times$ 8	11.4	0.0	0.0	0.0	3.8	10.8	4.6	5.0	5.8	6.0
Mixar ${}_{\text{stage1}}$	116M	32 $\times$ 32	15.2	3.8	2.0	2.2	2.4	8.6	3.8	4.2	4.6	4.7

Table 5: Scaling Mixar to 477M outperforms the English-only Pixar and text-based GPT-2 on average across multilingual generative tasks. For bAbI, we report few-shot free-generation accuracy (dash-delimited answers; 2 shots for the 477M model, 3 for the others) and for LAMBADA zero-shot last-word free-generation accuracy. Mixar generations are judged by a Qwen3.6-35B OCR judge with a pixel-exact shortcut and confusable folding, reading only the generated region on bAbI and the generation with a short anchoring context on LAMBADA, so that the judge never sees the question it is transcribing an answer to ( Appendix L ); the

stage2\text{-}gan

row is the adversarial stage-2 counterpart of the 8

\times

8 model ( Appendices J and K for training details). Pixar numbers are from Tai et al. (2024) .

Figure 3: Mixar can handle multilingual text as image , as shown here for examples of correct completions (black) for German, Spanish and Italian prompts (gray) on LAMBADA.

Figure 4: Mixar withstands English orthographic attacks more gracefully than tokenizer-based models : accuracy under increasing attack ratios on LAMBADA and bAbI (free-generation protocol, lookalike glyphs from the bundled fonts). Attack examples are shown in Fig. 6 .

Figure 5: An orthographic attack barely changes the rendered image. The same English LAMBADA prompt rendered before (left) and after (right) an attack at ratio $1.0$ , which replaces every attackable character by a Unicode lookalike. The text a tokenizer reads is almost entirely different, while the pixels Mixar reads are nearly unchanged.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

language	Letters outside the English alphabet
German	(Ä ä) (Ö ö) (SS ß) (Ü ü)
French	(À à) (Â â) (Æ æ) (Ç ç) (É é) (È è) (Ê ê) (Ë ë) (Î î) (Ï ï)(Ô ô) (Œ œ) (Û û) (Ù ù) (Ü ü) (Ÿ ÿ)
Spanish	(Á á) (Ch ch) (É é) (Í í) (Ll ll) (Ó ó) (Ú ú) (Ü ü)
Italian	(À à) (È è) (É é) (Ì ì) (Í í) (Î î) (Ò ò) (Ó ó)(Ù ù) (Ú ú)

Appendix

Table 6: This table shows the letters outside the English alphabet for other 4 latin languages of pretraining dataset.

Code	Language	Script	Pretraining
de	German	Latin	seen
en	English	Latin	seen
es	Spanish	Latin	seen
fr	French	Latin	seen
it	Italian	Latin	seen
ja	Japanese	Kana + Han	seen

Appendix

Table 7: Language codes used in the evaluation tables, with script and pretraining status. † : low-resource languages (class

\leq

2, Joshi et al., 2020 ).

Unicode block	PixeloidSans	Go Noto Current
Basic Latin (ASCII)	95/128	98/128
Latin-1 Supplement	117/128	96/128
Latin Extended-A	128/128	128/128
Latin Extended-B	97/208	208/208
Greek and Coptic	111/144	135/144
Cyrillic (+ Supplement)	174/304	304/304

Appendix

Table 8: Per-block character coverage (covered/total codepoints) of the two main bundled fonts of our rendering tool. The primary PixeloidSans font covers Latin scripts; Go Noto Current supplies the remaining scripts, including full kana and Hangul coverage and the common CJK ideographs.

Render Configuration		Model Structure		Stage 1 Hyperparameters
patch length	2	layers	12 / 24	peak lr	3e-4
patch number	360	attention heads	12 / 14	min. lr	3e-5
render DPI	80	hidden size	768 / 896	lr scheduler	CosineAnnealing
font size	32	activation	SwiGLU	optimizer	AdamW
patch size	32	intermediate size	3072 / 6656	$\beta_{1}$	0.9
font	PixeloidSans	parameters	116M / 477M	$\beta_{2}$	0.95

Appendix

Table 9: Rendering configuration, model structure and stage-1 pretraining hyperparameters of the 116M / 477M Mixar models. Stage-2 (GRPO) hyperparameters are reported separately in Table 11 .

Stage-2 objective (113M, 8 $\times$ 8)	steps	TF exact
none (stage-1 baseline)	–	5.1
GAN, adaptive $\lambda{=}0.1$ , threshold alt.	800	5.7
GAN, adaptive $\lambda{=}0.3$ , 1:1 alt.	800	5.6
GAN, adaptive $\lambda{=}0.3$ , 1:1 alt.	5000	7.4
GAN, adaptive $\lambda{=}0.3$ , 1:1 alt.	50000	6.9
GAN, adaptive $\lambda{=}0.3$ , threshold alt.	800	5.6

Appendix

Table 10: Stage-2 objective ablation for the 8

\times

8 Mixar : teacher-forced LAMBADA last-word exact-match (%, 2,500 passages, 5 languages). All GAN variants use the straight-through estimator through the discriminator and the adaptive gradient-ratio weight; without it every

\lambda_{m}\in\{1,3,9\}

diverges within 200 steps.

	113M (8 $\times$ 8)	116M (32 $\times$ 32)	477M (32 $\times$ 32)
initialization	GRPO 50k ckpt	4-generation chain	1-generation chain
batch / micro-batch	128 / 32	32 / 8	128 / 8
learning rate	$10^{-5}$	$5\times 10^{-6}$	$10^{-5}$
steps per generation	2000	1500–4000	2000
group size $G$	8	16	8
reward blend $d_{\text{blend}}$	0.5	0.7	0.5

Appendix

Table 11: Per-model rollout-GRPO hyperparameters of the released stage-2 models. The word gap is the all-white column run that marks a word boundary, calibrated to each font size.

Figure 6: Examples of 5 latin languages contains in pretraining dataset. 20% letters in these letters are changed by confused one.

stage1	MNLI	QQP	QNLI	SST-2	COLA	STSB	MRPC	RTE	WNLI
85M Pixar ${}_{\text{stage1}}$ lr	3e-5	3e-5	3e-5	3e-5	3e-5	3e-5	6e-5	3e-5	3e-5
116M Mixar ${}_{\text{stage1}}$ lr	3e-5	3e-5	3e-5	3e-5	3e-5	3e-5	6e-5	3e-5	6e-5
477M Mixar ${}_{\text{stage1}}$ lr	3e-5	3e-5	3e-5	3e-5	6e-5	3e-5	6e-5	3e-5	6e-5
Weight decay	0.1	0.1	0.1	0.01	0.01	0.01	0.01	0.01	0.01
Optimizer	AdamW
Warmup	Linear warmup

Appendix

Table 12: This table shows the hyperparameters applied in the GLUE benchmark.

Models	lr	Weight decay	Optimizer	Warmup	Warmup steps	$\beta_{1}$	$\beta_{2}$	Random seed	Batch size	Max steps	evaluation freq.
Mixar ${}_{\text{stage1}}$	3e-5	0.1	AdamW	Linear warmup	1000	0.9	0.95	42	256	8000	500
Mixar ${}_{\text{stage2}}$	3e-5	0.1	AdamW	Linear warmup	1000	0.9	0.95	42	256	8000	500

Appendix

Table 13: This table shows the hyperparameters used in XNLI downstream tasks.

Models	lr	Weight decay	Optimizer	Warmup	Warmup steps	$\beta_{1}$	$\beta_{2}$	Random seed	Batch size	Max steps	evaluation freq.
Mixar ${}_{\text{stage1}}$	5e-5	0.01	AdamW	Linear warmup	50	0.9	0.95	42	128	500	25

Appendix

Table 14: This table shows the hyperparameters used in SIB downstream tasks.

attack	LAMBADA				bAbI
ratio	GPT-2	MGPT-2	Mixar (116M)	Mixar (477M)	GPT-2	MGPT-2	Mixar (116M)	Mixar (477M)
0.0	32.6	26.2	9.2	13.4	41.4	38.0	16.6	34.6
0.01	30.0	24.6	9.0	12.6	43.2	36.2	17.0	34.6
0.05	19.0	15.6	8.8	13.2	35.8	24.2	15.0	23.8
0.1	8.4	7.2	7.6	9.6	23.0	14.0	13.2	18.6
0.2	1.6	2.2	4.8	7.4	8.0	4.2	11.2	11.4

Appendix

Table 15: Accuracy under the orthographic attack for each attack ratio (English LAMBADA and bAbI, free-generation protocol, lookalike glyphs restricted to the bundled fonts). The Mixar entries are the stage-2 models; MGPT-2, which shares our pretraining corpus, receives the identical attacked text. Prior-work numbers (e.g., Pixar ) are not comparable under this protocol and are omitted.

Models	Parameters	Patch size	MNLI-m/mm	QQP	QNLI	SST-2	COLA	STSB	MRPC	RTE	WNLI	AVG
Models	Parameters		392k	363k	108k	67k	8.5k	5.7k	3.5k	2.5k	635	AVG
PIXAR ${}_{\text{stage1}}$	85M	8 $\times$ 8	76.4 / 77.6	84.3	84.2	88.0	27.7	81.2	82.5	58.5	56.3	71.6
Mixar ${}_{\text{stage1}}$	116M	32 $\times$ 32	72.5 / 72.6	83.2	82.9	83.6	17.6	80.6	83.1	61.0	56.3	69.3
Mixar ${}_{\text{stage1}}$	474M	8 $\times$ 8	78.6 / 78.0	85.6	85.1	88.8	28.5	82.9	82.9	65.3	56.3	73.2
Mixar ${}_{\text{stage1}}$	477M	32 $\times$ 32	76.1 / 76.1	85.3	84.9	87.7	23.2	82.8	83.2	63.9	56.3	72.0

Appendix

Table 16: This table shows the comparison of Pixar and Mixar with the patch size of 8

\times

8 and 32

\times

32 trained on 0.1M steps and finetuned on GLUE tasks. Matthew’s correlation is used for COLA and Spearman’s

\rho

is applied for STSB. The F1 score is for MRPC and QQP. Other tasks used accuracy.

Dataset	Number of patches	chars/patch	GPT2 tok/patch
Chinese mc4	3,345,800,998	2.47	4.40
English mc4	17,387,543,604	3.26	0.81
French mc4	23,633,038,801	3.28	1.16
German mc4	28,550,835,190	3.20	1.19
Italian mc4	12,161,721,091	3.35	1.20
Spanish mc4	33,542,982,712	3.25	1.16

Appendix

Table 17: Patch-level data statistics for the Mixar pretraining languages in the multilingual mC4 dataset, including average characters and GPT-2 tokens per patch.

Figure 7: This image shows the comparison of patch size 8 and 32 for all pretraining scripts.

Figure 8: This image shows the correct samples of English, Chinese and Japanese bAbI tasks

Figure 9: This image shows the wrong samples of English, Chinese and Japanese bAbI tasks

Figure 10: This image shows some correct samples of LAMBADA task of eight pretraining languages

Figure 11: This image shows some wrong samples of LAMBADA task of eight pretraining languages

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Scaling Multilingual Pretraining of Pixel Language Models
Oct 21, 2026Ilker Kesen, Jenny Kunz, Kasra Malihi +1Multilingual Language Models Language Model Scaling
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen
Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models
Oct 16, 2026Haowen Tang, Wei Xia, Raoyuan Zhao +2Language Model Robustness LLM Fine-Tuning
Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)
Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Aug 31, 2026Ran Zhang, Miryam de Lhoneux, Wessel PoelmanContinued Pretraining Low-Resource Language Processing
LAGOM·NLP, Department of Computer Science, KU Leuven

Oct 21, 2026J/K move · Enter open · S save

Scaling Multilingual Pretraining of Pixel Language Models

Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott

Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen

Multilingual Language Models Language Model Scaling Cross-Lingual Transfer Cross-Lingual Representation Alignment

Open paper AskPDF arXiv

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.

Abstract

Multilingual Language Models

Explore similar work

Scaling Multilingual Pretraining of Pixel Language Models

Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation