Transformer-based autoregressive multilingual language models (LMs) are often assumed to store factual knowledge in an interlingua: a language-agnostic latent space. If this assumption holds, a parameter update applied in a source language during knowledge editing should inherently propagate to its semantic equivalents in target languages. However, empirical benchmarks show that cross-lingual propagation of edits often degrades substantially, a degradation that prior work treats largely as an algorithmic symptom rather than a structural bottleneck. In this work, we investigate whether representational anisotropy creates a structural bottleneck for source-only associative-memory editing, and perform a geometric dissection of the cross-lingual propagation gap. Using associative memory-based "locate-then-edit" methods, including ROME, MEMIT, and AlphaEdit, as analytical frameworks, we derive a closed-form decomposition of this gap into key-space propagation, parallel residual mismatch, and orthogonal residual mismatch. Our empirical findings show that cross-lingual propagation is primarily bottlenecked by a large orthogonal residual component, indicating that source-language edits often fail to span the target-language update directions required for successful transfer. Furthermore, we show that post-hoc correction using only source-language residual statistics encounters a large residual-prediction barrier. Finally, we perform a structural realignment via a targeted contrastive alignment objective and find that it substantially improves key-space propagation and partially improves cross-lingual edit transfer, while leaving value-space mismatch as a remaining bottleneck.
Figures & tables
zsRE (Edit Language: English)
CounterFact (Edit Language: English)
Method
English
German
Hindi
Urdu
Chinese
English
German
Hindi
Urdu
Chinese
ROME
76.12
82.77
63.09
61.47
51.29
50.78
72.62
49.61
46.70
33.59
MEMIT
27.37
19.12
19.94
15.41
2.20
5.15
13.02
7.09
8.65
0.87
AlphaEdit
75.58
59.29
36.64
38.81
36.64
96.99
89.61
58.74
54.08
57.86
Table 1: The cross-lingual transfer gap across editing methods in Llama-3.1-8B (ROME: layer 20; MEMIT and AlphaEdit: layers 16–20). Values represent Efficacy ( ↑ ): the proportion of cases where post-edit probability of the target answer exceeds that of the original answer. Native (English) and related (German) edits succeed, but transfer degrades sharply for typologically distant languages. Full evaluations, including Gemma-3-4B results, are detailed in Table 6 .
Figure 1: Key-space propagation ( α ) across editing methods (Llama-3.1-8B). The propagation magnitude follows a typologically stratified pattern. German ( αde ) peaks at early-to-mid layers, while distant languages are uniformly suppressed across ROME, MEMIT, and AlphaEdit. Corresponding Gemma-3-4B traces are shown in Figure 7 of Appendix D .
Figure 2: Key-space covariance geometry across layers (Llama-3.1-8B). Structural divergence is stratified by typological distance in the evaluated languages. (a) Frobenius divergence and (b) principal angles demonstrate that distant languages compress keys into misaligned, near-orthogonal subspaces ( >75∘ ). (c) Layer 20 heatmap confirms this misalignment is not English-centric. Gemma-3-4B results are detailed in Figure 6 of Appendix D .
Figure 3: Per-language gap decomposition across layers (ROME, Llama-3.1-8B). The relative total gap ∥g1,2∥/∥r1∥ (solid) is largely dominated by the orthogonal residual mismatch ∥r2⊥∥/∥r1∥ (dotted) in the evaluated language pairs. The parallel mismatch (dashed) acts as a visible secondary error only for typologically close languages (German). Gemma-3-4B results are shown in Figure 8 of Appendix D .
Figure 5Figure 6
α1,2↑
γ
∥r2⊥∥/∥r1∥↓
Efficacy ( ↑ )
Pair
Base
Aligned
Base
Aligned
Base
Aligned
Base
Aligned
en → de
0.212
0.858
0.470
0.486
0.913
0.861
72.62
61.17
en → hi
0.025
0.618
0.231
0.261
0.901
0.952
49.61
50.19
en → ur
0.015
0.517
0.197
0.223
0.882
0.941
46.70
59.22
en → zh
0.086
0.579
0.297
0.302
0.966
0.960
33.59
39.51
Table 4: Effect of contrastive alignment (Llama-3.1-8B, ROME, layer 20). The intervention dramatically improves key-space propagation ( α1,2 ), but the orthogonal value-space mismatch persists. Base Efficacy reflects CounterFact metrics from Table 1 . Results for other methods and Gemma-3-4B are presented in Tables 12 and 13 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Llama-3.1-8B
Gemma-3-4B
Shared
Hidden dim ( din )
14336
10240
Output dim ( dout )
4096
2560
Num layers
32
34
v∗ loss layer
31
33
v∗ learning rate
5×10−1
5×10−1
Appendix
Table 5: Hyperparameters for editing methods across models. Edit layers were selected by a sweep on CounterFact with English edits, scoring each candidate by Δ NLL (Definition 3) averaged over source and paraphrase prompts. ROME’s layer l∗ maximizes this score in English among nine candidate layers spanning the network. For MEMIT and AlphaEdit, the five-layer window S maximizes the score averaged over the four non-English languages among five to seven candidate windows spaced four layers apart, rather than being taken from localization results. The covariance C is the mean second moment of the keys over about 400,000 tokens of English Wikipedia.
Figure 6: The key-space geometry of Gemma-3-4B is governed by typological distance with a characteristic syntactic spike. (a) Frobenius divergence dF(Cen,Cj) remains relatively stable in early layers but increases steadily with layer depth, indicating the direction where internal key-space representations become most language-specific. (b) Principal angles between the bottom eigenspaces of English and target languages show that Indic languages (Hindi, Urdu) approach near-orthogonality ( >70∘ ) more rapidly than in Llama-3.1-8B. (c) Pairwise heatmap at layer 16 confirms that geometric misalignment is universal across all typologically distant pairs and is not restricted to English-centric transfers. The gap from English is much more prominent for Gemma-3-4B than Llama-3.1-8B.
Figure 7: Layer-wise trend for the mean key-space propagation scalar αˉ on Gemma-3-4B. The propagation magnitude follows a typologically stratified pattern, with the hierarchy de ≫ zh > hi ≈ ur . Compared to Llama-3.1-8B, the absolute magnitudes are generally lower, suggesting stronger cross-lingual key-space separation in Gemma-3-4B and weaker update propagation early in the forward pass.
Figure 8: Value-space gap decomposition for Gemma-3-4B. The relative total gap (solid line) is universally dominated by the orthogonal residual mismatch component r2⊥ (dotted line). Similar to the Llama-3.1-8B results, this directional misalignment accounts for the vast majority of the transfer failure, particularly at the optimal editing layers (28–32).
Figure 9: MEMIT Interference Ratio (IR) across layers. When editing in English, distant target languages suffer from severe off-diagonal cross-fact interference, corrupting the surviving update signal.
Figure 10: MEMIT Relative Batch Value Gap. The relative Frobenius gap ∥G1,2∥F/∥R1∥F remains consistently high, mimicking the single-edit ROME failure pattern across all target languages.
Figure 11: AlphaEdit Interference Ratio (IR) across layers. Despite the null-space projection, cross-fact interference remains heavily stratified by typological distance from English.
Figure 12: AlphaEdit Relative Batch Value Gap. The relative Frobenius gap ∥G1,2AE∥F/∥R1∥F confirms that orthogonal misalignment persists despite strict null-space parameter constraints.
Method
Model
Dataset
Edit Lang ↓ \ Eval Lang →
en
de
hi
ur
zh
ROME
Gemma3-4B
CounterFact
en
2.5839
0.6413
0.2455
0.0368
0.0251
de
2.0933
2.7755
0.0401
0.0009
-0.0265
hi
0.4531
0.3637
0.7743
0.0872
-0.0208
ur
0.1092
0.1264
0.0531
0.3397
-0.0277
zh
0.8474
0.5782
0.1661
0.0272
0.9829
zsRE
en
1.2839
0.4874
-0.0158
-0.0933
-0.0891
Appendix
Table 6: ΔNLLmargin across knowledge editing methods, architectures, and layer configurations. Values measure the change in the log-likelihood margin between the new target answer and the original answer, as defined in Appendix D.4 . Higher values indicate a greater shift toward the target answer.
Target
Fixed layer 20
Per-fact oracle
Improvement
German
0.864
0.774
0.090
Hindi
0.895
0.801
0.094
Urdu
0.872
0.775
0.097
Chinese
0.933
0.795
0.138
Appendix
Table 7: Per-fact oracle layer versus fixed layer 20 (ROME, Llama-3.1-8B, edit language English, 82 facts). Entries are ∥r2⊥∥/∥r1∥ (lower is better). The oracle picks, for each fact, the layer in {4,8,…,28} with the lowest value, using target-language information.
en →
1 token (49)
2 tokens (69)
3–4 tokens (188)
5+ tokens (76)
5+ minus 1 token
de
0.827
0.906
0.904
0.854
+0.027 ( p=0.28 )
hi
0.869
0.918
0.902
0.884
+0.015 ( p=0.49 )
ur
0.845
0.892
0.885
0.877
+0.032 ( p=0.21 )
zh
0.892
0.947
0.953
0.972
+0.080 ( p=0.02 )
Appendix
Table 8: Orthogonal mismatch ∥r2⊥∥/∥r1∥ by target length (zsRE, ROME, Llama-3.1-8B, layer 20, 382 facts). Bin sizes are in parentheses. The last column is the difference between the longest and shortest bins, with a Mann–Whitney test.
Condition
γ
∥r2⊥∥/∥r1∥
Difference from paraphrase
English paraphrase
0.556±0.039
0.718±0.031
–
English inversion
0.522±0.021
0.829±0.023
+0.111 ( p=4.9×10−3 )
German
0.591±0.019
0.826±0.020
+0.108 ( p=3.8×10−3 )
Urdu
0.190±0.013
0.920±0.044
+0.202 ( p=4.8×10−6 )
Hindi
0.229±0.013
0.941±0.047
+0.223 ( p=9.5×10−6 )
Chinese
0.400±0.023
1.031±0.058
+0.313 ( p=9.5×10−7 )
Appendix
Table 9: Logically equivalent and translated formulations of the same edited fact (21 facts). Differences are relative to the English paraphrase, with paired Wilcoxon signed-rank tests.
Figure 13: Estimated private fraction η^ across layers for keys (left), values (middle), and residuals (right) on LLaMA-3.1-8B. Values and residuals are both severely opaque ( η^≥0.9 ), confirming the value space as the fundamental bottleneck.
Layer
Target
Ridge
MLP
KNN
Best R2
PVT Frac.
4
de
− 0.0096
− 0.0109
− 0.0121
− 0.0096
1.0000
hi
− 0.0105
− 0.0103
− 0.0145
− 0.0103
1.0000
ur
− 0.0107
− 0.0117
− 0.0158
− 0.0107
1.0000
zh
− 0.0072
− 0.0075
− 0.0128
− 0.0072
1.0000
8
de
0.0002
− 0.0020
− 0.0045
0.0002
0.9998
hi
− 0.0053
− 0.0063
− 0.0093
− 0.0053
1.0000
Appendix
Table 10: Layer-wise R2 scores for each target language (source: English). Bold indicates the best estimator per row.
Hyperparameter
Value
Model & Architecture Details
Base Architecture
Llama-3.1-8B / Gemma-3-4B
Target Modules
up_proj , down_proj
Targeted Semantic Layers
16 – 25
Contrastive Alignment Objective
Training Epochs
10
Appendix
Table 11: Hyperparameters utilized during the cross-lingual contrastive alignment fine-tuning phase.
Figure 14: Evaluation perplexity and the optimal training window. Solid lines represent evaluation perplexity; dashed lines represent the pre-edit baseline. The optimal training window is highlighted by the stable region in Epochs 1–2 (Llama) and Epochs 1–3 (Gemma). Training beyond these points results in a rapid explosion of perplexity – most notably the Urdu ( ur ) peak in Gemma – signaling the onset of substantial degradation in language-modeling performance.
Method
Model
Layers
en (Aligned / Base)
de (Aligned / Base)
hi (Aligned / Base)
ur (Aligned / Base)
zh (Aligned / Base)
ROME
Gemma-3-4B
Layer 16
0.5466 / 0.4615
0.3682 / 0.2053
0.0129 / -0.0813
-0.0883 / -0.1102
-0.0948 / -0.1403
Layer 20
0.3437 / 1.0960
0.1776 / 0.4939
0.0016 / 0.1400
-0.0334 / 0.0477
0.0055 / 0.0244
Layer 24
0.5908 / 1.1212
0.3594 / 0.6356
0.0232 / 0.1119
-0.0190 / 0.0086
0.0032 / 0.0401
Llama-3.1-8B
Layer 16
1.0365 / -0.8460
0.7196 / -0.1831
-0.0186 / -0.2192
0.0403 / -0.2403
-0.3892 / -0.5233
Layer 20
1.4522 / 0.2435
1.1788 / 0.3758
-0.0012 / -0.0200
0.0577 / -0.0344
-0.1535 / -0.1970
Layer 24
1.8763 / -0.3794
1.4756 / 0.2568
0.0365 / 0.0119
0.0559 / -0.0011
-0.0682 / -0.0497
Appendix
Table 12: ΔNLLmargin score comparison between Base and Aligned models on CounterFact. Higher positive values indicate greater success in updating factual knowledge.
Method
Model
Layers
en (Aligned / Base)
de (Aligned / Base)
hi (Aligned / Base)
ur (Aligned / Base)
zh (Aligned / Base)
ROME
Gemma-3-4B
Layer 16
61.55 / 71.17
58.45 / 61.26
43.88 / 50.97
40.78 / 48.25
41.94 / 45.24
Layer 20
68.25 / 84.66
60.87 / 73.69
51.46 / 59.71
47.48 / 56.31
51.75 / 55.24
Layer 24
74.95 / 83.01
68.35 / 78.25
46.02 / 62.91
48.06 / 47.86
49.32 / 56.70
Llama-3.1-8B
Layer 16
56.70 / 39.71
56.31 / 56.12
56.41 / 44.37
59.32 / 36.21
38.35 / 25.63
Layer 20
63.59 / 50.78
61.17 / 72.62
50.19 / 49.61
59.22 / 46.70
39.51 / 33.59
Layer 24
69.42 / 39.90
66.99 / 74.27
61.65 / 50.87
63.50 / 43.88
50.68 / 40.00
Appendix
Table 13: Efficacy score comparison on CounterFact. Values represent update success (Efficacy ↑ ), defined as the proportion of cases where the post-edit probability of the target answer exceeds that of the original answer.
Model
Lang
Aligned
Base
Llama-3.1-8B
en
0.45
0.50
de
0.43
0.48
hi
0.41
0.47
ur
0.37
0.45
zh
0.38
0.42
Appendix
Table 14: XNLI evaluation scores for Llama-3.1-8B before (Base) and after (Aligned) finetuning across languages.
Renmin University of China · Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · School of Computing, National University of Singapore +1