Hundreds of languages have written text but almost no translations, and whether a model learning one from untranslated text builds on a related language or familiar letters is unclear: they usually coincide. Here we show that familiar letters, not a related language, let an unseen language into a shared space of meaning. GlyphMind, a tokenizer-free vision transformer that reads text as pixels and lets any sentence be shown in any script, was pretrained to align translations across 27 writing systems. Adapted on untranslated Lao, a language and script it had never seen, it retrieved English translations seven times as often as the strongest simple baseline. Rewritten in unfamiliar letters, Lao did not enter although its close relative Thai was in pretraining; Santali, which had none, entered in Devanagari, familiar from Hindi, but not in its own script. Transliteration into familiar scripts may thus open multilingual models to languages without translations.
Figures & tables
Figure 1: Familiar letters let unseen languages into GlyphMind’s shared space. a , GlyphMind reads each sentence as one line image of 64×768 pixels, here in Thai. A vision transformer with no tokenizer turns the image’s 8×8 -pixel patches into one sentence vector. In training, translations are pulled together and masked spans are predicted. b , The shared space, drawn schematically. Each dot is a sentence, and translations lie close together. The cube shows a small part of the space. The lens enlarges the cluster for the first sentence of Article 1 of the Universal Declaration of Human Rights. c , Lao letters resemble the Thai letters seen in training. An unfamiliar version redraws each Lao letter as a character of the Vai syllabary. Santali’s own script, Ol Chiki, is absent from training. A familiar version transliterates Santali into Devanagari, which the model reads from Hindi. Santali in Devanagari thus uses the same letters as Hindi, whereas Lao letters only resemble Thai ones. Arrows point from each language’s own script to the version made from it. d , Top-5 retrieval of the English translation after adaptation on matched amounts of untranslated text. Lao sentences are ranked against 1,997 candidates and Santali sentences against 1,012, so chance is below 0.5%. Coloured and hollow bars show three runs adapted on text in the same script as the test sentences. Grey bars show nine runs adapted on text in the other three scripts. Bars show the mean and lines show the range. Colour marks the home region of each script. Blue is Europe, ochre West Asia and Africa, vermilion South Asia, teal Southeast Asia and purple East Asia. Hollow grey marks scripts absent from training.
Figure 2: GlyphMind’s shared space spans many scripts. a , Retrieval from each pretraining language into English on NTREX-128. Each sentence is compared with 1,997 English sentences. Bars show how often its translation ranks first. They are ordered by accuracy and coloured by the home region of each script, as in Fig. 1. Each language is also named in its own script. The dashed line marks the median of the 21 languages. Chance is 0.05%. b , Retrieval accuracy from a against the number of pretraining lines in the writing system of each language, on a logarithmic axis (Supplementary Table 1). Armenian had the fewest lines. c , Retrieval from Russian into English for GlyphMind-205M and two untrained networks. One is a copy of GlyphMind-205M, randomly initialized, and the other is a small convolutional network with random weights that sees only image texture.
Figure 3: A script absent from pretraining enters the shared space. a , Retrieval from Lao into English on NTREX-128 after adaptation on about one million untranslated Lao lines. Each Lao sentence is compared with 1,997 English sentences. Bars show how often its translation ranks among the first five. The model before adaptation and the strongest of eight simple baselines are shown for comparison. b , The same model on 997 Lao sentences from the FLORES-200 dev split, held out from every choice of adaptation settings. c , Two GlyphMind-38M models pretrained on the same data with and without Thai, scored on NTREX-128 before either saw any Lao text. Each model was trained once. Error bars in a and b are 95% confidence intervals. Dashed lines mark chance, 0.25% in a and c and 0.5% in b .
Figure 4: The advantage of familiar letters persists in checks on ink, visible text and shared digits and words. All panels show how often the English translation ranks among the first five, without example translations. a , Ordinary Lao on NTREX-128, drawn with lighter strokes, thinner strokes or a lighter font so that it carries about as little ink as the redrawn Lao. Ink is measured on the test lines relative to normal renders. Below each condition, the first word of a Lao sentence is drawn the same way. b , Santali in Devanagari on FLORES-200, scored on whole lines and on only the text that each line shows in Ol Chiki. c , Santali in Devanagari with digits and words shared with the pretraining text removed, on FLORES-200 and IN22-Gen. Bars show means over runs, and lines in b show their range.
Extended Data Fig. 1 ∣ What GlyphMind is given. Each row is one model input, a greyscale image of 64 by 768 pixels. The English row in a shows the whole image and its 8 by 8-pixel patches, and the other rows are cropped to the band that holds ink. Text is drawn at the largest size that fits, down to about 28 pixels, and longer text is cut at a word or character boundary. a , The first sentence of Article 1 of the Universal Declaration of Human Rights in seven pretraining languages. Renders of one meaning group serve as positives for one another. b , Variants made from the English line during pretraining. The cipher positive replaces each letter with a new random shape. The augmented copy is shifted, rotated and given noise. c , Hard negatives change the text before it is rendered. Shading marks the changed words. d , One Lao and one Santali sentence in the four versions of the experiment. Ordinary Lao is drawn in three fonts, and each redrawn font replaces the Lao letters with Vai characters. Coloured cells mark familiar letters, as in Fig. 1c. The cipher shapes, noise level, negatives, Vai letters and Devanagari spelling are examples made for this figure.
Extended Data Fig. 2 ∣ The objectives of pretraining and adaptation. a , Lao-English geometry agreement before any Lao text, against Russian-to-English top-1 retrieval on NTREX-128, at successive checkpoints of pretraining. GlyphMind-205M learns translation and span prediction. The GlyphMind-38M model trained on the same data with Thai learns translation alone. Agreement uses pure renders and the pooled backbone (Methods). The two models also differ in size and learning-rate schedule, and checkpoints of GlyphMind-205M five epochs apart differ by up to 0.1. The dashed line is a randomly initialized GlyphMind-205M. b , Adaptation objectives compared in preliminary experiments on a smaller model (Methods). Each run started from the same checkpoint and trained on 48,084 lines of Lao Wikipedia. Agreement is read at the projection head and at the pooled backbone, on NTREX-128 pure renders.
Extended Data Fig. 3 ∣ Training curves of adaptation. a , The reported Lao run of GlyphMind-205M, scored every three epochs on NTREX-128 pure renders. The optimizer state was reset after epoch 12, and the final checkpoint, epoch 24, is reported. With 500 example translations, scores average three draws of the pairs. b , Russian-to-English top-1 retrieval on unmodified renders during the same run. c , Every run of the four-version experiment, scored every ten epochs on its own test set, NTREX-128 for Lao and FLORES-200 devtest for Santali. Lines show the three runs adapted on each version, and a dashed line marks a run that replayed the pretraining mixture. With 500 example translations, scores average ten draws. The grey bar and dot give the range and mean of the nine runs adapted on the other versions at epoch 80. Dashed horizontal lines show the model before adaptation. Every run is reported at epoch 80, as fixed in advance (Methods).
Extended Data Fig. 4 ∣ Retrieval with example translations. a , Lao-to-English retrieval of the reported Lao run on NTREX-128 with 500 example translations, scored on 800 held-out pairs and averaged over three draws. The controls are a randomly initialized GlyphMind-205M and example translations paired at random. b , The same measure against the number of example translations, for GlyphMind-205M before and after adaptation, as mean and s.d. over ten draws. c , In the preliminary model, a rotation fitted on about 1,000 pairs in 512 dimensions scores far higher on its own pairs than on held-out pairs.
Geometry agreement
Lao retrieval, top-5 (%)
Russian (%)
Checkpoint
head
backbone
500 examples
direct
top-1
Before adaptation
0.231
0.262
7.58
3.41
56.5
After 12 epochs
0.242
0.241
16.21
8.71
46.3
Final, 12 epochs after the reset
0.217
0.180
21.25
14.27
44.4
Randomly initialized
0.181
0.184
1.21
0.65
0.25
Extended Data Table 1 ∣ GlyphMind-205M before and after adaptation on Lao. The rows show GlyphMind-205M before adaptation, after 12 epochs on 991,775 Lao lines, and at the final checkpoint of the reported Lao run, 12 epochs after a reset of the optimizer state. The last row is a randomly initialized GlyphMind-205M. Another random initialization gives 0.33% Russian retrieval (Fig. 2c). Geometry agreement measures whether Lao and English sentences are arranged alike, with sentence length controlled, on NTREX-128 pure renders (Methods). It is read at the projection-head output and at the pooled backbone. Lao retrieval uses NTREX-128 pure renders. With 500 example translations, scores average three draws of the pairs and rank 800 held-out candidates. Direct retrieval ranks all 1,997 English sentences. Russian retrieval uses unmodified renders. Bold marks the reported Lao run.
With Thai
Without Thai
Epoch
10
20
30
10
20
30
Retrieval into English (%)
Thai, top-1, unmodified renders
32.3
40.2
42.0
0.55
0.60
0.30
Thai, top-1, pure renders
25.5
32.2
35.0
0.20
0.25
0.15
Trained-script skill, top-1
28.5
35.7
37.5
28.6
33.9
35.5
Lao, direct, top-5, pure renders
3.86
3.46
1.55
0.60
0.65
0.60
Extended Data Table 2 ∣ Removing Thai from pretraining. Two GlyphMind-38M models were trained on the same mixture, with and without its Thai renders, and scored at epochs 10, 20 and 30 (Methods). They are compared at epoch 10, where their trained-script skill matches. Trained-script skill is the mean top-1 retrieval into English for Russian, Greek, Hindi and Georgian, four languages in both mixtures. All measures use NTREX-128, except that Ol Chiki uses FLORES-200. Pure renders keep only the characters of a script’s own Unicode block. Geometry agreement is read at the pooled backbone unless a row says otherwise. Chance is 0.05% for top-1 and 0.25% for top-5 retrieval. The strongest simple baseline reaches 2.05% in Lao direct retrieval.
Ordinary Lao
Redrawn Lao
Santali, Devanagari
Devanagari, matched
Santali, Ol Chiki
Model
ex
dir
ex
dir
ex
dir
ex
dir
ex
dir
GlyphMind-205M before adaptation
7.03
3.41
2.20
0.35
8.76
3.36
9.96
3.46
2.56
0.79
Randomly initialized GlyphMind-205M
1.18
0.65
1.14
0.35
1.04
0.59
1.28
0.59
0.82
0.49
Adapted on ordinary Lao
seed 0 a
16.40
9.46
3.63
0.25
9.86
3.66
9.78
4.15
3.82
0.89
seed 1
16.19
9.86
3.06
0.40
11.96
4.25
10.48
4.55
2.38
0.69
Extended Data Table 3 ∣ The four-version experiment, every model on every test set. Top-5 retrieval into English, in per cent, at epoch 80 of every run. Lao is tested on NTREX-128 and Santali on FLORES-200 devtest. ex, retrieval with 500 example translations, averaged over ten draws and scored on 800 held-out pairs for Lao and 500 for Santali. dir, direct retrieval against all 1,997 English sentences for Lao and 1,012 for Santali. Bold marks the three runs adapted on the version being tested. The matched test set keeps only the characters that each Ol Chiki test line shows, transliterated into Devanagari (Methods). Chance is 0.63% for Lao and 1.0% for Santali with example translations, and 0.25% and 0.49% in direct retrieval. The strongest simple baseline in direct retrieval reaches 2.05% for Lao and 3.36% for Santali. On their own test set, the three runs average 15.7 (ex) and 9.4 (dir) for ordinary Lao and 3.2 and 0.38 for redrawn Lao. They average 14.0 and 6.8 for Santali in Devanagari, 13.7 and 5.5 on the matched set, and 6.4 and 1.2 for Santali in Ol Chiki. On the same test sets, the nine runs adapted on the other versions average 8.6 and 3.0, 3.0 and 0.29, 9.7 and 3.4 (9.6 and 3.7 matched), and 3.8 and 0.95. a These runs replayed the pretraining mixture. All other runs replayed its FLORES-200 variant (Methods).
Check
Result
How test lines are drawn
NTREX-128 Lao
median font size 28 px, 90% at 40 px or less. 73% of lines cut, and a line shows a median 56% of its characters
FLORES-200 Santali, Ol Chiki
28 px. 100% of lines cut, and a line shows 34% of its characters
FLORES-200 Santali, Devanagari
28 px. 94% of lines cut, and a line shows 53% of its characters
Matched Devanagari test set
lead over the nine other runs 1.9 points, with all three Devanagari runs above all nine others, against 3.4 on whole lines
Letters identified by a template matcher, 30 noisy copies of each (%, ordinary / redrawn Lao)
Extended Data Table 4 ∣ Checks on alternative explanations. Each block reports one check described in Methods. Lao is drawn in Noto Sans Lao unless a row names another font. Retrieval is into English, top-5 and direct unless a row says otherwise. In the ink block, the three values are the mean of the three ordinary-Lao runs, the model before adaptation and the reported Lao run. In the Devanagari block and the Ol Chiki row, they are the mean of the three runs adapted on that version, the mean of the nine other runs and the model before adaptation. Gains are over the model before adaptation. Shared words occur three or more times in the Devanagari lines of the pretraining mixture, its FLORES-200 variant or the Bible corpus. Leads are in percentage points, with 95% intervals from 2,000 bootstrap resamples of the test lines, paired across models (Methods).
Writing system
Language
Lines
Bible
Latin
English
4,077,098
0
Cyrillic
Russian; Ukrainian (Bible)
491,966
30,959
Greek
Greek; Koine Greek (Bible)
475,873
26,415
Armenian
Armenian
4,690
0
Georgian
Georgian
205,687
0
Hebrew
Hebrew
477,980
30,966
Table 1: *
Setting
Value
Rendering
Image
64 by 768 pixels, greyscale
Font size
starts at 0.7 times the image height and shrinks until the text fits, down to 0.44 times the height, about 28 pixels
Text that does not fit
cut at a word or character boundary
Layout
RAQM shaping where available
Augmentation in training
shift of up to 2 pixels, rotation of up to 2 degrees, Gaussian noise
Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.
Muge Zhang, Aaron Jencks, Krishna Badikela +2
Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.
Ran Zhang, Miryam de Lhoneux, Wessel Poelman
LAGOM·NLP, Department of Computer Science, KU Leuven
Vision-language models score well on mathematical, scientific, and spatial reasoning benchmarks, yet these evaluations are overwhelmingly English. I present the first cross-lingual visual reasoning audit for Indian languages. 980 questions from MathVista, ScienceQA, and MMMU are translated into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi using IndicTrans2, with Gemini 2.0 Flash cross-verification on 50 samples per language (inter-translator agreement 0.79-0.84). Eight VLMs, from 7B open-source models to GPT-4o, are evaluated across all seven languages, yielding 68,600 inference records that include text-only and chain-of-thought ablations. I find accuracy drops of 9.8-25 percentage points when switching from English to an Indian language, with Dravidian languages suffering up to 13.2 pp more than Indo-Aryan. Chain-of-thought prompting degrades Bengali (-14.4 pp) and Kannada (-11.4 pp) rather than helping, exposing English-centric reasoning chains. Aya-Vision-8B, built for 23 languages, still drops 28.5 pp on Dravidian scripts; multilingual pretraining alone does not transfer visual reasoning. I release the translated benchmark and all model outputs.
Swastik R
Indian Institute of Information Technology, Raichur