Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops
Authors: Faeze Ghorbanpour, Constanza Fierro, Alexander Fraser, Anders Søgaard
Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML) · Department of Computer Science, University of Copenhagen
Large Language Models (LLMs) often answer the same factual question differently across languages. We study whether cross-lingual latent-space intervention can reduce this inconsistency. We train layer-specific autoencoders on parallel multilingual representations and apply inference-time corrections to factual QA prompts. We find that latent intervention improves geometric alignment between languages, and that this improvement translates into consistent gains in cross-lingual consistency with English across both open-ended and multiple-choice QA formats, without degrading factual accuracy. In open-ended QA, Spearman's rank correlation between English and non-English languages improves substantially, with gains of 0.16 for English-Arabic and 0.20 for English-Russian pairs. In multiple-choice QA, answer agreement with English improves consistently across both KLAR and mParaRel. Ablations show that AE reconstruction yields consistent gains at no accuracy cost, while PCA projection contributes marginally, and mean-shift produces substantially larger consistency gains in open-ended QA at the cost of some accuracy.
Figures & tables
KLAR OE
KLAR MC
mParaRel MC
Intervention
Acc
Rank Corr (en)
Acc
Agr (en)
Acc
Agr (en)
Raw
No intervention
56.77
6.68
85.59
85.09
85.91
86.70
+ Mean-shift
53.78
14.09
85.59
85.01
81.09
76.02
+ PCA
56.72
6.81
85.56
84.97
85.86
86.29
AE
No intervention
56.40
7.77
85.32
85.43
86.35
86.97
+ Mean-shift
53.45
24.91
85.42
85.41
85.91
86.68
Table 1: Average accuracy (%) and consistency with English across strategies at the best layer per setting. Rank Corr (en) is the average Spearman rank correlation ( ×100 ) across English-paired language pairs. Agr (en) is the average answer agreement (%) with English. Bold indicates best result per column.
KLAR OE Rank Corr
KLAR MC Agreement
mParaRel MC Agreement
ar-en
en-nl
en-ru
en-zh
ar-en
en-nl
en-ru
en-zh
ar-en
en-nl
en-ru
en-zh
Raw
2.52
16.21
4.55
3.45
66.17
94.65
90.87
88.66
84.31
90.06
87.33
85.09
AE
2.51
17.83
4.93
5.82
66.36
95.23
91.22
88.89
84.80
90.55
87.43
85.19
+ PCA
3.77
18.52
5.85
4.84
66.44
95.19
91.22
88.85
84.80
90.45
87.43
85.19
+ Mean-shift
18.17
38.27
24.35
18.86
66.82
95.23
91.18
88.81
84.30
90.90
86.71
85.80
Table 2: Pairwise English consistency before and after intervention at the best layer per setting. Rank Corr values are Spearman rank correlation ( ×100 ). Agreement values are answer agreement (%).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: t-SNE visualizations of FLORES+ parallel sentence representations in the raw LLM space (left) and AE latent space (right) at layers 20, 24, and 28. Colors indicate languages.
Figure 2: First row: Cosine similarity to English before and after latent-space intervention across layers for open-ended and multiple-choice QA (KLAR and mParaRel). Second row: First-token accuracy (%) before and after latent-space intervention across layers for open-ended and Multiple-choice QA (KLAR and mParaRel). Dashed lines show before intervention; solid lines show after intervention.
Variant
Cosine
P@1
Raw LLM space
67.61
38.36
w/o self-reconstruction loss
13.92
0.84
w/o alignment loss
76.02
83.22
Full objective
86.51
95.46
Appendix
Table 3: Loss ablation on FLORES+ with Llama-3.1-8B. Cosine is the average cosine similarity between parallel sentence pairs; P@1 is parallel-sentence retrieval accuracy (%).
Open-ended QA
Multiple-choice QA
Method
ar-en
en-nl
en-ru
en-zh
Avg
Acc
ar-en
en-nl
en-ru
en-zh
Avg
Acc
Raw
21.70
38.55
19.46
6.84
21.64
45.85
48.76
89.58
87.06
79.84
76.31
78.21
PCA
24.27
37.95
19.31
7.31
22.21
45.48
48.76
89.69
87.40
79.92
76.44
78.25
Mean-shift
22.98
36.93
20.75
4.32
21.25
42.31
50.63
88.85
85.76
80.26
76.38
78.21
AE
20.71
40.29
20.96
8.84
22.70
49.00
50.13
89.77
87.74
80.41
77.01
78.55
+ PCA
24.48
41.18
21.61
6.61
23.47
48.51
50.36
89.73
87.86
80.18
77.03
78.59
Appendix
Table 4: Llama-3.1-8B on KLAR. Open-ended values are Spearman rank correlation ( ×100 ) with English; multiple-choice values are answer agreement (%) with English. Acc is factual accuracy (%). Bold indicates best result per column.
Open-ended QA
Multiple-choice QA
Method
ar-en
en-nl
en-ru
en-zh
Avg
Acc
ar-en
en-nl
en-ru
en-zh
Avg
Acc
Raw
14.60
26.93
3.97
-0.92
11.15
37.89
68.61
94.50
92.21
91.22
86.64
87.51
PCA
16.09
27.46
4.29
0.89
12.18
37.33
69.07
94.46
92.36
91.33
86.81
87.60
Mean-shift
1.92
20.58
7.33
12.95
10.70
27.50
72.70
94.35
91.79
90.68
87.38
88.25
AE
15.31
29.90
4.98
-0.84
12.34
37.55
69.03
93.97
92.33
91.56
86.72
87.45
+ PCA
14.38
29.27
4.65
0.52
12.21
37.47
68.92
94.04
92.36
91.49
86.70
87.44
Appendix
Table 5: Qwen-3-8B on KLAR. Open-ended values are Spearman rank correlation ( ×100 ) with English; multiple-choice values are answer agreement (%) with English. Acc is factual accuracy (%). Bold indicates best result per column.
Open-ended QA
Multiple-choice QA
Layer
ar-en
en-nl
en-ru
en-zh
Avg
Acc
ar-en
en-nl
en-ru
en-zh
Avg
Acc
Raw
21.70
38.55
19.46
6.84
21.64
45.85
48.76
89.58
87.06
79.84
76.31
78.21
Two-phase
L20
18.71
38.29
20.96
8.84
21.70
49.00
50.82
88.35
86.75
81.37
76.82
78.12
L24
21.56
39.29
19.03
5.78
21.42
48.89
50.13
89.77
87.74
80.41
77.01
78.55
L28
22.16
37.53
17.97
6.14
20.95
49.14
49.37
89.00
87.13
81.02
76.63
77.99
L31
17.92
34.55
17.51
7.76
19.44
40.75
43.26
85.76
80.64
73.77
70.86
71.35
Appendix
Table 6: Two-phase versus joint training of the autoencoder and language-specific decoders, evaluated on KLAR with Llama-3.1-8B. Open-ended values are Spearman rank correlation ( ×100 ) with English; multiple-choice values are answer agreement (%) with English.
Cross-lingual consistency in language models studies whether equivalent inputs across languages produce comparable knowledge, reasoning, and safety behavior.