cs.CLOct 18, 2026
SaveWhere Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
Organizations: Department of Linguistics, University of Illinois Urbana-Champaign
Abstract
Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Figures & tables
Figure 1: Language-specific clusters for the Europarl Corpus in layer 8 (left) and layer 16 (right) of Llama 3.2 1B. Each color represents a different language: Portuguese (blue), Spanish (orange), and French (green). The x, y, and z axes display the raw cosine distances from the Spanish, French, and Portuguese centroids, respectively.
| Portuguese | Spanish | French | ||||||||||||||||
| L | Nearest centroid (%) | Mean distances | Nearest centroid (%) | Mean distances | Nearest centroid (%) | Mean distances | ||||||||||||
| PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | |
| 1 | 99 | 0.8 | 0.5 | -1.2 | 0.2 | 0.9 | 0.0 | 99 | 0.0 | 0.2 | -1.1 | 0.9 | 0.0 | 0.0 | 99 | 0.8 | 0.7 | -1.4 |
| 8 | 99 | 1.5 | 0.8 | -1.1 | 0.1 | 1.1 | 0.1 | 98 | 0.0 | 0.1 | -1.0 | 0.9 | 0.2 | 0.1 | 99 | 0.8 | 0.5 | -1.5 |
| 16 | 99 | 0.0 | 0.0 | -1.2 | 0.3 | 0.9 | 0.3 | 99 | 0.2 | 0.3 | -1.2 | 0.9 | 0.5 | 0.1 | 99 | 0.8 | 0.7 | -1.4 |
Table 1: Distinctiveness of language-specific clusters in the Europarl Corpus. Portuguese , Spanish , and French at the top refer to the language-specific clusters; thus, the first column represents how close Portuguese sentences are to other Portuguese sentences relative to their parallel equivalents. The L column refers to the layer. The values under Mean distances indicate the actual distance to each centroid, standardized relative to the global mean and standard deviation in that layer. Thus, a value of -1.2 would indicate that Portuguese sentences in layer one are more than one standard deviation closer to other Portuguese sentences than to their parallel counterparts. Both measures indicate a consistent level of language-specific information within each layer.
Figure 2: Distribution of ambiguous and non-ambiguous strings in layer 8. The blue, orange, and green clusters represent Portuguese, Spanish, and French, respectively. The x, y, and z axes display the raw cosine distances from the Spanish, French, and Portuguese centroids in the Europarl corpus, respectively. The outlined red diamonds represent the ambiguous sentences, located between the Portuguese and Spanish clusters, enlarged in the red box.
| Ambiguous | Portuguese | Spanish | French |
| Este documento será publicado esta semana. [ This document will be published this week. ] | Comuniquei-o ontem a respeito disso. Este documento será publicado esta semana. [ I informed him about this yesterday. This document will be published this week. ] | No hay escapatoria. Este documento será publicado esta semana. [ There’s no way out. This document will be published this week. ] | Il n’y a aucune issue. Este documento será publicado esta semana. [ There is no way out. This document will be published this week. ] |
Table 2: Example from the Ambiguous String Corpus. The first column contains an ambiguous string that shows language membership ambiguity between Portuguese and Spanish. The second, third, and fourth columns exemplify the structure of non-ambiguous strings with priming in Portuguese, Spanish, and French, respectively. French was chosen to represent the neutral condition given that it does not display language membership ambiguity with the other two languages. In the non-ambiguous condition, the priming structure is highlighted in red; the rest of the string is identical to the ambiguous sentence. English translations are provided between square brackets. More examples are found in Appendix D.
| Portuguese | Spanish | French | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| L | PT | SP | FR | AB | PT | SP | FR | AB | PT | SP | FR | AB |
| 1 | 100.0 | 8.45 | 5.63 | 84.51 | 0.00 | 91.55 | 78.87 | 15.49 | 0.00 | 0.00 | 15.50 | 0.00 |
| 8 | 100.0 | 1.41 | 0.00 | 77.46 | 0.00 | 98.59 | 0.00 | 22.54 | 0.00 | 0.00 | 100.0 | 0.00 |
| 16 | 2.82 | 0.00 | 0.00 | 0.00 | 0.00 | 98.59 | 0.00 | 32.39 | 97.18 | 1.41 | 100.0 | 67.61 |
Table 3: Ambiguous String Corpus: bold Portuguese , Spanish , and French at the top of each section refer to the language clusters. The L column refers to layer number. The upper section contains the percentages of strings in each language that are closer to the Portuguese, Spanish, and French centroids, respectively. The lower section contains their average distance to each centroid. Distances are standardized.
Figure 3: Mean change in distances to the target language centroid caused by priming. The x-axis lists all layers; the y-axis represents the average difference in distances to the centroids. The blue, orange, and green lines represent non-ambiguous Portuguese, Spanish, and French, respectively.
Figure 4: Mean weighted cosine scores of priming-induced movement in the top 100 language-informative embedding dimensions. The x-axis lists all layers; the y-axis represents the mean weighted cosine values. The blue, orange, and green lines represent the movement for Portuguese, Spanish, and French, respectively. Values are standardized.
| L | Por. | Spa. | Galician | Asturian | English | German | French | Italian | Hindi | Thai | Macro Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.93 | 0.93 | 0.88 | 0.88 | 0.93 | 0.97 | 0.96 | 0.91 | 0.74 | 1.00 | 0.91 |
| 2 | 0.93 | 0.94 | 0.85 | 0.86 | 0.92 | 0.98 | 0.97 | 0.92 | 0.74 | 1.00 | 0.90 |
| 3 | 0.93 | 0.95 | 0.86 | 0.86 | 0.92 | 0.98 | 0.98 | 0.94 | 0.75 | 1.00 | 0.90 |
| 4 | 0.94 | 0.96 | 0.87 | 0.85 | 0.92 | 0.97 | 0.98 | 0.95 | 0.77 | 0.99 | 0.90 |
| 5 | 0.94 | 0.96 | 0.87 | 0.84 | 0.92 | 0.98 | 0.98 | 0.95 | 0.76 | 0.99 | 0.89 |
| 6 | 0.94 | 0.97 | 0.86 | 0.85 | 0.92 | 0.97 | 0.98 | 0.95 | 0.76 | 0.99 | 0.88 |
Table 4: F-Scores across model layers by language. Higher F-scores indicate that embeddings contain more language-specific information because the classifier is able to make a distinction between languages across many samples using only sentence embeddings. Spanish and Portuguese are most distinctive, thus having more language-specific information, in the middle and final layers.
| Nearest centroid (%) | Mean distances | Nearest centroid (%) | Mean distances | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | ||||
| 1 | PT | 99.90 | 0.09 | 0.00 | -1.21 | 0.21 | 0.80 | 2 | PT | 99.99 | 0.00 | 0.00 | -1.10 | 0.08 | 0.77 |
| SP | 0.82 | 99.16 | 0.02 | 0.21 | -1.18 | 0.73 | SP | 3.45 | 96.53 | 0.01 | 0.12 | -1.06 | 0.62 | ||
| FR | 0.52 | 0.03 | 99.45 | 0.98 | 0.92 | -1.48 | FR | 2.36 | 0.01 | 97.63 | 1.14 | 0.97 | -1.56 | ||
| 3 | PT | 99.97 | 0.03 | 0.00 | -1.21 | 0.35 | 0.77 | 4 | PT | 99.92 | 0.08 | 0.00 | -1.29 | 0.43 | 0.84 |
| SP | 0.78 | 99.20 | 0.02 | 0.34 | -1.09 | 0.52 | SP | 0.24 | 99.74 | 0.03 | 0.38 | -1.12 | 0.52 | ||
Appendix A . Organization of parallel sentences across all layers for the Europarl Corpus. The values under Nearest centroids refer to the number of parallel sentences closer to each centroid in relation to all the others (percent). If Spanish sentences are closest to the Spanish centroid, this indicates a language-specific organization. The values under Mean distances indicate the actual distance to each centroid, standardized relative to the global mean and standard deviation in that layer. Thus, a value of -1.21 would indicate that Portuguese sentences in layer one are more than one standard deviation closer to other Portuguese sentences than to their parallel counterpart in other layers. Both measures indicate a consistent level of language-specific information within each layer.
| Nearest centroid (%) | Mean distances | Nearest centroid (%) | Mean distances | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | ||||
| 1 | PT | 99.35 | 0.63 | 0.02 | -1.18 | 0.20 | 0.78 | 2 | PT | 98.18 | 1.74 | 0.08 | -1.03 | 0.15 | 0.64 |
| SP | 1.16 | 98.73 | 0.12 | 0.19 | -1.12 | 0.68 | SP | 0.96 | 98.58 | 0.46 | 0.12 | -0.91 | 0.51 | ||
| FR | 0.76 | 0.66 | 98.58 | 1.00 | 0.91 | -1.48 | FR | 1.54 | 6.09 | 92.38 | 1.06 | 0.98 | -1.53 | ||
| 3 | PT | 96.85 | 2.77 | 0.38 | -1.14 | 0.45 | 0.63 | 4 | PT | 97.14 | 2.72 | 0.15 | -1.27 | 0.52 | 0.76 |
| SP | 0.16 | 94.77 | 5.08 | 0.35 | -0.94 | 0.39 | SP | 0.07 | 98.80 | 1.13 | 0.41 | -1.02 | 0.42 | ||
Appendix C . Organization of parallel sentences across all layers for the TED Corpus. The values under Nearest centroids refer to the number of parallel sentences closer to each centroid in relation to all the others (percent). If Spanish sentences are closest to the Spanish centroid, this indicates a language-specific organization. The values under Mean distances indicate the actual distance to each centroid, standardized relative to the global mean and standard deviation in that layer. Thus, a value of -1.18 would indicate that Portuguese sentences in layer one are more than one standard deviation closer to other Portuguese sentences than to their parallel counterpart in other layers. Both measures indicate a consistent level of language-specific information within each layer.
Appendix B . Language-specific clusters for the TED Corpus in layer 8 (left) and layer 16 (right) of Llama 3.2 1B. Each color represents a different language: Portuguese (blue), Spanish (orange), and French (green). The x, y, and z axes display the raw cosine distances from the Spanish, French, and Portuguese centroids, respectively.
| Ambiguous | Portuguese | Spanish | French |
|---|---|---|---|
| Este documento será publicado esta semana. [ This document will be published this week. ] | Comuniquei-o ontem a respeito disso. Este documento será publicado esta semana. [ I informed him about this yesterday. This document will be published this week. ] | No hay escapatoria. Este documento será publicado esta semana. [ There’s no way out. This document will be published this week. ] | Il n’y a aucune issue. Este documento será publicado esta semana. [ There is no way out. This document will be published this week. ] |
| Vi algo importante esta semana. [ I saw something important this week. ] | Comuniquei-o ontem a respeito disso. Vi algo importante esta semana. [ I informed him about this yesterday. I saw something important this week. ] | No hay escapatoria. Vi algo importante esta semana. [ There’s no way out. I saw something important this week. ] | Il n’y a aucune issue. Vi algo importante esta semana. [ There’s no way out. I saw something important this week. ] |
| Sabíamos que era pesado, este período. [ We knew it was tense, this period. ] | Comuniquei-o ontem a respeito disso. Sabíamos que era pesado, este período. [ I informed him about this yesterday. We knew it was tense, this period. ] | No hay escapatoria. Sabíamos que era pesado, este período. [ There’s no way out. We knew it was tense, this period. ] | Il n’y a aucune issue. Sabíamos que era pesado, este período. [ There’s no way out. We knew it was tense, this period. ] |
| Miguel avisa que construirá armas de guerra nucleares. [ Miguel warns that he will build nuclear war weapons. ] | Comuniquei-o ontem a respeito disso. Miguel avisa que construirá armas de guerra nucleares. [ I informed him about this yesterday. Miguel warns that he will build nuclear war weapons. ] | No hay escapatoria. Miguel avisa que construirá armas de guerra nucleares. [ There’s no way out. Miguel warns that he will build nuclear war weapons. ] | Il n’y a aucune issue. Miguel avisa que construirá armas de guerra nucleares. [ There’s no way out. Miguel warns that he will build nuclear war weapons. ] |
| Miguel continua implementando políticas públicas. [ Miguel continues to implement public policies. ] | Comuniquei-o ontem a respeito disso. Miguel continua implementando políticas públicas. [ I informed him about this yesterday. Miguel continues to implement public policies. ] | No hay escapatoria. Miguel continua implementando políticas públicas. [ There’s no way out. Miguel continues to implement public policies. ] | Il n’y a aucune issue. Miguel continua implementando políticas públicas. [ There’s no way out. Miguel continues to implement public policies. ] |
| Miguel está prolongando problemas políticos. [ Miguel is prolonging political problems. ] | Comuniquei-o ontem a respeito disso. Miguel está prolongando problemas políticos. [ I informed him about this yesterday. Miguel is prolonging political problems. ] | No hay escapatoria. Miguel está prolongando problemas políticos. [ There’s no way out. Miguel is prolonging political problems. ] | Il n’y a aucune issue. Miguel está prolongando problemas políticos. [ There’s no way out. Miguel is prolonging political problems. ] |
Appendix D . Examples from the Ambiguous Strings Corpus. The first column contains an ambiguous string that shows language membership ambiguity between Portuguese and Spanish. The second, third, and fourth columns exemplify the structure of non-ambiguous strings with priming in Portuguese, Spanish, and French, respectively. French was chosen to represent the neutral condition given that it does not display language membership ambiguity with the other two languages. In the non-ambiguous condition, the priming structure is highlighted in red; the rest of the string is identical to the ambiguous string. English translations are provided between square brackets. Each row represents a different grammatical construction used to ensure variation in the strings.
| Nearest centroid (%) | Mean distances | Nearest centroid (%) | Mean distances | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PT | SP | FR | PT | SP | FR | PT | SP | FR | PT | SP | FR | ||||
| 1 | PT | 100.0 | 0.00 | 0.00 | -1.38 | -0.29 | 1.58 | 2 | PT | 100.0 | 0.00 | 0.00 | -1.50 | 0.11 | 1.35 |
| SP | 8.45 | 91.55 | 0.00 | -0.64 | -1.04 | 1.56 | SP | 60.56 | 39.44 | 0.00 | -0.90 | -0.81 | 1.53 | ||
| FR | 5.63 | 78.87 | 15.50 | 0.17 | -0.15 | 0.23 | FR | 0.00 | 54.93 | 45.07 | 0.30 | -0.02 | 0.06 | ||
| AB | 84.51 | 15.49 | 00.0 | -0.87 | -0.48 | 1.32 | AB | 98.59 | 1.41 | 0.00 | -1.12 | -0.43 | 1.44 | ||
| 3 | PT | 100.0 | 0.00 | 0.00 | -1.77 | 0.48 | 1.35 | 4 | PT | 100.0 | 0.00 | 0.00 | -1.93 | 0.43 | 1.54 |
Appendix E . Organization of parallel sentences across all layers for the Ambiguous Strings Corpus. The values under Nearest centroids refer to the number of parallel sentences closer to each centroid in relation to all the others (percent). If Spanish sentences are closest to the Spanish centroid, this indicates a language-specific organization. The values under Mean distances indicate the actual distance to each centroid, standardized relative to the global mean and standard deviation in that layer. Thus, a value of -1.38 would indicate that Portuguese sentences in layer one are more than one standard deviation closer to other Portuguese sentences than to their parallel counterpart in other layers. Ambiguous and non-ambiguous sentences are indicated in the second column: Non-Ambiguous Portuguese (PT), Non-Ambiguous Spanish (SP), Non-Ambiguous French (FR), and Ambiguous (AB) which could be Spanish or Portuguese.
More in this topic
Multilingual Language Models
Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.
See all 256 papers