One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations

Sripad Karne

cs.CLMar 9, 2026

One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations

Authors: Sripad Karne

Organizations: Columbia University

Abstract

Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.

Figures & tables

Comparison Type	Mean Jaccard Similarity
Cross-Script Original (Latin $\leftrightarrow$ Cyrillic, same sentence)	0.58
Cross-Script Paraphrase (Latin $\leftrightarrow$ Cyrillic, same paraphrase)	0.59
Cross-Script Cross-Paraphrase (Latin/Cyrillic Orig $\leftrightarrow$ Cyrillic/Latin Para)	0.47
Cross-Script Random (Latin $\leftrightarrow$ Cyrillic, unrelated)	0.28
Cross-Language Random (Serbian $\leftrightarrow$ English, unrelated)	0.19

Table 1: Evidence for script-invariant representations. Results averaged across all models and layers.

Figure 1: Baseline semantic discrimination across model scales. Larger models achieve greater separation between paraphrase and random similarity, with Serbian Latin and Serbian Cyrillic following nearly parallel trajectories.

Figure 2: Cross-script similarity across model scales. Cross-script original and paraphrase similarity increase with model size while cross-script random decreases, indicating increasingly robust script-invariant representations. Notably, cross-script cross-paraphrase remains stable across scales.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Mean token counts by script and sentence type.

Figure 4: Token count difference vs. SAE feature Jaccard similarity.

Figure 5: LaBSE sentence similarity verification. (a) Histogram of similarity scores showing clear separation between random pairs and semantically related pairs. (b) Box plots of similarity distributions by condition. Cross-script original pairs achieve near-ceiling similarity, confirming semantic equivalence across scripts.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings
May 29, 2026Sripad KarneLLM Interpretability Sparse Autoencoders
Columbia University
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
SharedSAE: One Feature Dictionary Across Language Models
Sep 3, 2026Daniil Ognev, Célian Vasson, Lijie Hu +2LLM Interpretability Sparse Autoencoders
MBZUAI · Sorbonne Université · Tohoku University +1

May 29, 2026J/K move · Enter open · S save

How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings

Sripad Karne

Columbia University

LLM Interpretability Sparse Autoencoders Automated Evaluation

Open paper AskPDF arXiv

Sparse autoencoder (SAE) features are increasingly used to interpret language models, with auto-generated natural-language labels serving as the primary interface for understanding what each feature represents. We ask whether these labels generalize: does a feature labeled for a concept actually track that concept across languages and scripts? Using Serbian digraphia as a controlled testbed -- the same language written in both Latin and Cyrillic via deterministic transliteration -- we first find that SAE feature sets activated by the same content in different languages, scripts, and wordings share substantial overlap (mean Jaccard 0.39 vs 0.13 random baseline, peaking at 0.57), suggesting genuine cross-lingual semantic features. We then test whether auto-interpretation labels keep pace. They often do not: features whose labels describe semantic content miss the same meaning in Serbian up to 4

\times

more often than within English, and miss Serbian Cyrillic more than Serbian Latin -- two scripts that are deterministic transliterations of each other. The gap grows with network depth, yet the labels give no indication that they fail. These results suggest that auto-interpretation labels reflect a feature's behavior on the languages and scripts a model has seen most in training, rather than the concept itself.

Abstract

LLM Interpretability

Explore similar work

How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

SharedSAE: One Feature Dictionary Across Language Models