The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.
Figures & tables
Figure 1: Geometric properties of final-layer token representations in Llama 3.1 8B (see Figures 5 and 5 in the appendix for all models). Lower-resource languages exhibit greater representational degeneration.
Figure 2: Pairwise cosine similarity of representations across model layers, calculated as specified in Section 3.1.1 and averaged across languages within each tier of data availability.
Model
0%
25%
50%
75%
100%
Cosine similarity
Llama 3.2 1B
-0.52
-0.84
-0.89
-0.88
-0.88
Llama 3.2 3B
-0.24
-0.87
-0.89
-0.85
-0.90
Llama 3.1 8B
-0.52
-0.93
-0.93
-0.90
-0.95
Qwen3 1.7B
-0.56
-0.76
-0.85
-0.82
-0.73
Qwen3 4B
-0.56
-0.78
-0.91
-0.85
-0.74
Table 1: Spearman ρ between MADLAD size and geometric metrics across layers (0% = input embeddings, 100% = final layer, remaining columns are evenly-spaced hidden layers). We boldface ρ<−0.5 for cosine similarity and ρ>0.5 for IsoScore.
Δ Cosine similarity
Δ IsoScore
Model
CPT
+CR
+IS
CPT
+CR
+IS
Llama 3.2 1B
-.096
-.278
-.089
.000
-.004
+.014
Llama 3.2 3B
-.023
-.176
-.024
.000
-.003
+.022
Llama 3.1 8B
-.020
-.202
-.049
.000
-.003
+.027
Qwen3 1.7B
-.105
-.620
-.048
.000
-.002
+.004
Qwen3 4B
-.095
-.491
-.032
.000
-.002
+.006
Table 2: Final-layer geometry change from base models to CPT and regularised CPT (+CR: cosine regularisation, +IS: I-STAR), averaged over 10 CPT languages.
Model
Variant
AfriXNLI
AfriMMLU
AfriMGSM
Overall
0-shot
5-shot
0-shot
5-shot
0-shot
8-shot
Avg
Δ
Δ %
Llama 3.2 1B
Base
32.9
32.4
27.1
25.8
2.1
2.7
20.5
–
–
CPT
34.0
33.5
26.1
26.6
2.2
2.5
20.8
+0.3
+1.5%
+ CosReg
33.4
33.2
26.5
26.6
2.3
2.6
20.8
+0.3
+1.5%
+ I-STAR
33.9
33.2
26.0
26.5
2.2
2.6
20.7
+0.2
+1.0%
Gemma 3 1B
Base
33.4
32.7
24.9
25.8
1.7
2.7
20.2
–
–
Table 3: Average performance (%) across CPT languages (Base = pretrained model, CPT = vanilla continued pretraining, +CosReg / +I-STAR = regularised CPT). AfriMGSM scores are the averages of direct and chain-of-thought evaluations. We boldface the best-performing variant per model. Δ and Δ % show, respectively, absolute and relative performance gains over base models.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Code
Language
Words
Characters
hau
Hausa
151,868,683
858,393,671
amh
Amharic
86,144,742
453,397,345
sot
Sesotho
35,331,080
193,620,015
zul
isiZulu
31,584,011
271,658,420
yor
Yorùbá
30,854,371
155,028,472
ibo
Igbo
29,890,153
161,814,604
Appendix
Table 4: Corpus sizes for CPT (word counts are based on white-space separated tokens in the raw text).
Figure 3: Isotropy (IsoScore) of representations across model layers, calculated as specified in Section 3.1.2 and averaged across languages within each tier of data availability.
Figure 4: The relationship between pretraining data availability and average pairwise cosine similarity of final-layer representations. Each dot represents a single language.
Figure 6: How do CPT and geometric regularisation change cosine similarity? We plot layer-wise cosine similarity, averaged across all 10 languages from our CPT experiments, comparing base models to vanilla CPT models (which only marginally affect cosine similarity), and geometrically regularised CPT models (CosReg successfully reduces final-layer cosine similarity, while leaving other layers unaffected).
Figure 7: How do CPT and geometric regularisation change isotropy? We plot layer-wise IsoScore, averaged across all 10 languages from our CPT experiments, comparing base models to vanilla CPT models (which only marginally affect isotropy), and geometrically regularised CPT models (I-STAR successfully increases final-layer IsoScore, while leaving other layers unaffected).
Model
Variant
kin
hau
amh
xho
sna
zul
ibo
yor
sot
orm
Avg
kin
hau
amh
xho
sna
zul
ibo
yor
sot
orm
Avg
AfriXNLI 0-shot (%)
AfriXNLI 5-shot (%)
Llama 3.2 1B
Base
33.3
34.5
32.8
32.3
31.3
33.0
32.7
34.2
33.7
30.8
32.9
31.7
33.7
30.7
31.7
29.2
32.3
32.7
34.7
32.8
32.5
32.4
CPT
33.3
34.8
33.0
33.5
33.7
36.2
33.8
33.8
33.3
33.5
34.0
34.3
33.7
33.8
34.2
31.3
35.5
33.2
32.8
33.8
33.0
33.5
+ CosReg
33.7
34.5
32.7
33.3
34.5
33.0
33.5
33.0
31.5
33.3
33.4
33.2
35.0
34.2
33.7
31.2
35.7
32.5
32.2
32.7
32.5
33.2
+ I-STAR
33.3
34.7
32.5
33.2
33.5
34.2
33.7
33.5
35.2
33.5
33.9
33.0
34.7
33.5
33.2
31.5
34.7
33.8
33.3
32.7
31.7
33.2
Llama 3.2 3B
Base
33.2
34.2
33.2
33.3
33.3
34.5
33.8
33.0
33.5
34.0
33.6
33.3
33.7
34.3
31.8
30.8
33.7
31.8
29.2
33.0
33.2
32.3
Appendix
Table 5: AfriXNLI and AfriMMLU performance (%) across CPT languages. Best variants boldfaced .
Model
Variant
kin
hau
amh
xho
sna
zul
ibo
yor
sot
orm
Avg
kin
hau
amh
xho
sna
zul
ibo
yor
sot
orm
Avg
AfriMGSM 0-shot (%)
AfriMGSM 8-shot (%)
Llama 3.2 1B
Base
2.0
2.4
2.8
3.2
2.0
2.0
1.6
2.8
2.0
2.0
2.2
2.4
2.8
1.6
1.6
2.0
2.4
2.8
3.6
2.4
2.4
2.5
CPT
3.2
3.6
2.4
2.4
2.0
2.0
0.8
1.2
2.0
2.0
2.1
2.0
4.4
1.2
3.2
2.8
2.8
2.4
2.0
2.8
1.6
2.7
+ CosReg
2.8
3.2
2.8
3.6
1.6
2.4
1.2
1.2
2.0
2.0
2.2
2.4
4.0
1.2
3.2
2.8
3.6
2.4
1.6
3.2
2.0
2.8
+ I-STAR
3.2
3.6
3.2
2.8
1.6
2.4
0.8
1.6
2.4
2.0
2.3
2.0
4.0
1.2
3.2
2.4
2.8
2.4
1.6
3.2
2.0
2.6
Llama 3.2 3B
Base
2.4
4.0
3.2
3.2
3.2
3.2
3.6
4.4
2.4
2.8
3.2
4.8
5.6
2.4
5.6
3.6
4.4
2.8
4.8
5.6
2.8
4.4
Appendix
Table 6: AfriMGSM direct and chain-of-thought performance (%) across CPT languages. Best variants boldfaced .