Are Language Models Script-Aware?

David Kletz; Sandra Mitrović; Ljiljana Dolamić; Fabio Rinaldi

cs.CLOct 6, 2026

Are Language Models Script-Aware?

Authors: David Kletz, Sandra Mitrović, Ljiljana Dolamić, Fabio Rinaldi

Organizations: SUPSI, IDSIA, Switzerland · armasuisse, Science & Technology, Switzerland

Abstract

Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model's response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%) and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.

Figures & tables

Figure 2: Script forcing results. Scores represent the percentage of cases where the script of the generated text matched the requested output script. Axes represent language-script pairs formatted as L-S, where L indicates the output language (S: Serbian, F: French, E: English, K: Kazakh) and S indicates the script (L: Latin, C: Cyrillic, A: Arabic). The language in parentheses indicates the prompt language.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Lang-script	Total words	Types	Sentences
Kz-ar	27,805,939	810,230	57,827
Kz-cyr	4,656,620	301,614	371,902
Kz-lat	430	371	100
Sr-cyr	15,255	4,424	150
Sr-lat	569,206	89,537	2,919
Cr-cyr	517,595	109,847	77,688

Appendix

Table 1: Summary statistics of datasets

Figure 3: Size of the vocabulary (number of non-character tokens) per language-script combination, for each model. (with Zipf)

Config	gpt-4o-m			gpt-4o			Deepseek-chat			Haiku			Sonnet			Gemini-flash			Gemini
Config	F	C	S	F	C	S	F	C	S	F	C	S	F	C	S	F	C	S	F	C	S
F-C vs F-L	-4	0	-28	-8	-4	-56	-16	-14	-99	-17	-18	-75	-18	-17	-50	-7	-4	-73	-13	-16	-61
S-C vs S-L (S-L)	-3	-9	-10	-2	-5	0	-4	-4	0	-2	-8	9	-6	-8	0	-12	-15	0	0	-6	-3
S-C vs S-L (S-C)	-8	-11	-60	-6	-9	0	-4	-5	-1	6	1	-5	-4	-6	0	-6	-10	-1	-1	-2	-36
S-C vs S-L (E)	6	5	-51	2	-1	-1	0	4	0	-9	-10	0	-3	-4	0	-24	-29	-44	-10	-8	-44
K-C vs K-A (K)	-21	-13	-8	-15	-9	-6	4	5	-1	-9	7	-7	-2	4	-1	10	9	-2	-5	1	-1

Appendix

Table 2: Quality score differences across script configurations of LLMs. F: Factuality, C: Completeness, S: Script fidelity. Lines name are language-script pairs formatted as L-S, where L indicates the output language (S: Serbian, F: French, E: English, K: Kazakh) and S indicates the script (L: Latin, C: Cyrillic, A: Arabic). The language in parentheses indicates the prompt language.

Config	DeepSeek-R1-Distill-Llama-8B			Llama-2-7B-Chat			Mistral-7B-Instruct			Mixtral-8x7B-Instruct
Config	F	C	S	F	C	S	F	C	S	F	C	S
F-C vs F-L	5	2	-99	-1	-2	-100	-4	2	-84	-4	-7	-93
S-C vs S-L (S-L)	0	-4	-42	-5	-3	-34	2	0	68	-2	-3	-36
S-C vs S-L (S-C)	2	2	-63	-4	-2	-99	-11	-10	-99	4	8	-95
S-C vs S-L (E)	1	4	-96	-1	1	-99	-11	-10	-63	2	-2	-89
K-C vs K-A (K)	3	6	-26	0	0	-55	0	0	-85	10	7	11

Appendix

Table 3: Quality score differences across script configurations of SLMs. F: Factuality, C: Completeness, S: Script fidelity. Lines name are language-script pairs formatted as L-S, where L indicates the output language (S: Serbian, F: French, E: English, K: Kazakh) and S indicates the script (L: Latin, C: Cyrillic, A: Arabic). The language in parentheses indicates the prompt language.

Panel of Figure 4 — Figure 4: Perplexity distributions across script configurations for each SLM. Each panel shows perplexity values for different language-script combinations, with the language of the prompt in parenthesis.

Language Script Type	Number of Languages
Mono-scriptic	78
Multi-scriptic	10
Mixed	9

Appendix

Table 4: Number of languages in different script-based categories based on Tatoeba and literature. Mono-scriptic (multi-scriptic) are those languages for which both sources agree that they use a single (multiple) script(s), respectively. Mixed category languages use different number of scripts (mono in Tatoeba and multi in the literature, or vice versa).

Mixed Category	Multi-script
	Category
Bulgarian	Bosnian
Kurdish (Central)	Chavacano
Javanese	Kunming
Korean	Kazakh
Marathi	Kurmanji

Appendix

Table 5: Left: Mixed category languages according to Tatoeba and literature (using different number of scripts in Tatoeba with respect to the literature). Right: Multi-script category containing languages which both Tatoeba and literature consider multi-scriptic.

Source URL	Language
https://sites.google.com/site/seesscm/languages-in-multiple-scripts	Bosnian
	Bulgarian
	Kurdish (Central)
	Macedonian
	Moldavian
	Tajik

Appendix

Table 6: Literature sources (left) explicitly mentioning multi-scriptic feature of a language (right).

Panel of Figure 5 — Figure 5: Percentage of languages by script usage. Blue bars represent languages for which the model generates text in a single script, while orange bars represent languages for which the model uses two or more scripts. The input groups correspond to the language categories defined in Sub-Section D.1 .

Panel of Figure 6 — Figure 6: Distribution of output scripts for Latin (right column) and Cyrillic (left column) input prompts for open-source SLMs. Lighter colors indicate higher frequency

Panel of Figure 7 — Figure 7: Distribution of output scripts for Latin (right column) and Cyrillic (left column) input prompts for proprietary LLMs. Lighter colors indicate higher frequency

Nb	Language prompt	Script prompt	Language Question	Requested language output	Requested script output
1	French	Latin	French	French	Latin
2	French	Latin	French	French	Cyrillic
3	English	Latin	French	French	Latin
4	English	Latin	Kazakh	Kazakh	Cyrillic
5	English	Latin	Kazakh	Kazakh	Latin
6	English	Latin	Kazakh	Kazakh	Arabic

Appendix

Table 7: Detail of configurations for the script forcing experiment (see Section 2 ).

Model Type	Adaptation	Forcing
Mixtral (Quantized)	$\sim$ 15 hrs	$\sim$ 5 hrs
Other SLMs	$\sim$ 7 hrs	$\sim$ 2 hrs
LLMs (via API)	$\sim$ 1-5 hr	$\leq$ 1-3 hrs

Appendix

Table 8: Approximate execution times per 1,000 prompts

Model	Adaptation	Forcing
GPT-4o-mini	$\sim$ $0.5	$\sim$ $1
GPT-4o	$\sim$ $3	$\sim$ $10
DeepSeek-Chat	$\sim$ $0.5	$\sim$ $2
Claude-3.5-Haiku	$\sim$ $1.5	$\sim$ $5
Claude-Sonnet-4	$\sim$ $5	$\sim$ $20
Gemini-Flash	$\sim$ $1	$\sim$ $3

Appendix

Table 9: Approximate API costs for each experimental configuration.

Explore similar work

The Latin Substrate: How Language Models Represent and Mediate Script Choice
May 29, 2026Daniil Gurgurov, Alan Saji, Katharina Trinley +2Representational Capacity Choice
D German Research Center for AI (DFKI) · I Indian Institute of Technology, Madras · H University of Hamburg +1
Script Choice in LLMs: Evidence for Late-Layer Commitment
Sep 23, 2026David Kletz, Sandra Mitrović, Itay Sabato +2Cryptographic Merkle Tree Commitments Commitment
SUPSI, IDSIA, Switzerland · Independent Researcher · armasuisse, Science & Technology, Switzerland
The Digital Afterlife of Empires: Four Language Models Converge on the Same Imperial Cartography of Writing
Apr 30, 2026Hiroki FukuiWriting Asymmetry
Research Institute of Criminal Psychiatry / Sex Offender Medical Center · Department of Neuropsychiatry, Kyoto University

May 29, 2026, cs.CLJ/K move · Enter open · S save

The Latin Substrate: How Language Models Represent and Mediate Script Choice

Daniil Gurgurov, Alan Saji, Katharina Trinley, Josef van Genabith+1

D German Research Center for AI (DFKI) · I Indian Institute of Technology, Madras · H University of Hamburg+1

Representational Capacity Choice

Open paper AskPDF arXiv

Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely preserving semantic content. The vector generalizes to writing systems unseen during construction: it flips non-Latin output to Latin, but maps Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the model. Overall, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.

Abstract

Explore similar work

The Latin Substrate: How Language Models Represent and Mediate Script Choice

Script Choice in LLMs: Evidence for Late-Layer Commitment

The Digital Afterlife of Empires: Four Language Models Converge on the Same Imperial Cartography of Writing