Familiar letters, not related languages, let unseen languages into a shared space of meaning
Organizations: Huazhong University of Science and Technology, Wuhan, China
Abstract
Hundreds of languages have written text but almost no translations, and whether a model learning one from untranslated text builds on a related language or familiar letters is unclear: they usually coincide. Here we show that familiar letters, not a related language, let an unseen language into a shared space of meaning. GlyphMind, a tokenizer-free vision transformer that reads text as pixels and lets any sentence be shown in any script, was pretrained to align translations across 27 writing systems. Adapted on untranslated Lao, a language and script it had never seen, it retrieved English translations seven times as often as the strongest simple baseline. Rewritten in unfamiliar letters, Lao did not enter although its close relative Thai was in pretraining; Santali, which had none, entered in Devanagari, familiar from Hindi, but not in its own script. Transliteration into familiar scripts may thus open multilingual models to languages without translations.
Figures & tables
| Geometry agreement | Lao retrieval, top-5 (%) | Russian (%) | |||
|---|---|---|---|---|---|
| Checkpoint | head | backbone | 500 examples | direct | top-1 |
| Before adaptation | 0.231 | 0.262 | 7.58 | 3.41 | 56.5 |
| After 12 epochs | 0.242 | 0.241 | 16.21 | 8.71 | 46.3 |
| Final, 12 epochs after the reset | 0.217 | 0.180 | 21.25 | 14.27 | 44.4 |
| Randomly initialized | 0.181 | 0.184 | 1.21 | 0.65 | 0.25 |
| With Thai | Without Thai | |||||
| Epoch | 10 | 20 | 30 | 10 | 20 | 30 |
| Retrieval into English (%) | ||||||
| Thai, top-1, unmodified renders | 32.3 | 40.2 | 42.0 | 0.55 | 0.60 | 0.30 |
| Thai, top-1, pure renders | 25.5 | 32.2 | 35.0 | 0.20 | 0.25 | 0.15 |
| Trained-script skill, top-1 | 28.5 | 35.7 | 37.5 | 28.6 | 33.9 | 35.5 |
| Lao, direct, top-5, pure renders | 3.86 | 3.46 | 1.55 | 0.60 | 0.65 | 0.60 |
| Ordinary Lao | Redrawn Lao | Santali, Devanagari | Devanagari, matched | Santali, Ol Chiki | ||||||
| Model | ex | dir | ex | dir | ex | dir | ex | dir | ex | dir |
| GlyphMind-205M before adaptation | 7.03 | 3.41 | 2.20 | 0.35 | 8.76 | 3.36 | 9.96 | 3.46 | 2.56 | 0.79 |
| Randomly initialized GlyphMind-205M | 1.18 | 0.65 | 1.14 | 0.35 | 1.04 | 0.59 | 1.28 | 0.59 | 0.82 | 0.49 |
| Adapted on ordinary Lao | ||||||||||
| seed 0 a | 16.40 | 9.46 | 3.63 | 0.25 | 9.86 | 3.66 | 9.78 | 4.15 | 3.82 | 0.89 |
| seed 1 | 16.19 | 9.86 | 3.06 | 0.40 | 11.96 | 4.25 | 10.48 | 4.55 | 2.38 | 0.69 |
| Check | Result |
|---|---|
| How test lines are drawn | |
| NTREX-128 Lao | median font size 28 px, 90% at 40 px or less. 73% of lines cut, and a line shows a median 56% of its characters |
| FLORES-200 Santali, Ol Chiki | 28 px. 100% of lines cut, and a line shows 34% of its characters |
| FLORES-200 Santali, Devanagari | 28 px. 94% of lines cut, and a line shows 53% of its characters |
| Matched Devanagari test set | lead over the nine other runs 1.9 points, with all three Devanagari runs above all nine others, against 3.4 on whole lines |
| Letters identified by a template matcher, 30 noisy copies of each (%, ordinary / redrawn Lao) | |
| Writing system | Language | Lines | Bible |
|---|---|---|---|
| Latin | English | 4,077,098 | 0 |
| Cyrillic | Russian; Ukrainian (Bible) | 491,966 | 30,959 |
| Greek | Greek; Koine Greek (Bible) | 475,873 | 26,415 |
| Armenian | Armenian | 4,690 | 0 |
| Georgian | Georgian | 205,687 | 0 |
| Hebrew | Hebrew | 477,980 | 30,966 |
| Setting | Value |
|---|---|
| Rendering | |
| Image | 64 by 768 pixels, greyscale |
| Font size | starts at 0.7 times the image height and shrinks until the text fits, down to 0.44 times the height, about 28 pixels |
| Text that does not fit | cut at a word or character boundary |
| Layout | RAQM shaping where available |
| Augmentation in training | shift of up to 2 pixels, rotation of up to 2 degrees, Gaussian noise |
| Run | Initial epoch | Learning rate | Replay | Epochs | ex | dir |
|---|---|---|---|---|---|---|
| Lao sweep, 991,775 lines | ||||||
| Configuration 1 | 20 | 0.7 | 12 | 11.08 | 4.91 | |
| Configuration 2 | 40 | 0.7 | 12 | 10.62 | 6.21 | |
| Configuration 3 | 20 | 0.7 | 12 | 14.50 | 7.46 | |
| Configuration 4 | 40 | 0.7 | 12 | 14.13 | 8.21 | |
| Configuration 5 | 20 | 0.7 | 12 | 15.33 | 7.36 | |
Multilingual Language Models
Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.
See all 278 papers