Cross-lingual Representation Learning via Centroid Intervention Fusion

Wei Sun; Marie-Francine Moens

cs.CLAug 26, 2026

Cross-lingual Representation Learning via Centroid Intervention Fusion

Authors: Wei Sun, Marie-Francine Moens

Organizations: Department of Computer Science, KU Leuven Celestijnenlaan 200A 3001 Heverlee, Belgium

Abstract

Large language models (LLMs) exhibit uneven multilingual performance, especially when dealing with low-resource languages. Inference-time intervention offers a lightweight way to improve cross-lingual transfer by modifying the hidden states produced by the LLMs during the forward pass, without updating model parameters. However, existing cross-lingual intervention methods typically learn separate projections from source to target languages, which limits scalability and prevents knowledge sharing across languages. We propose Centroid Intervention Fusion (CIF), a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator. Across multilingual commonsense reasoning, natural language inference, factual editing, and machine translation benchmarks, CIF outperforms the strongest prior pairwise intervention baseline by up to +3.378 pp on average across four model backbones, while supporting performance gains for low resource languages. The code is available at https://github.com/VRCMF/CIF.git.

Figures & tables

Figure 1: Comparison between existing pairwise cross-lingual intervention methods and CIF. Existing methods use separate intervention operators for each language pair, while CIF unifies separate operators into a single language-shared operator via centroid fusion.

Figure 2: Overview of Centroid Intervention Fusion (CIF). CIF first learns language-specific projection matrices from multilingual parallel corpora and converts them into intervention perturbations around the identity transformation. It then computes a centroid perturbation in the projection weight space to construct a unified language-shared intervention matrix. During inference, the fused intervention matrix is applied to hidden representations of inputs to enable stable and transferable cross-lingual intervention across languages.

Method	XCOPA	XStoryCloze	XWinograd	XCSR	XNLI	MZSRE	WMT	Flores
Zero-shot	53.17	81.39	43.61	39.12	56.06	39.98	39.57	38.45
Translation-NLLB	59.41	76.29	52.45	44.02	49.49	42.81	23.60	32.81
Intervention Methods
ITI	55.20 $\pm$ 0.28	73.05 $\pm$ 0.66	35.60 $\pm$ 1.03	36.18 $\pm$ 0.14	55.83 $\pm$ 0.31	39.28 $\pm$ 0.22	36.58 $\pm$ 0.57	38.23 $\pm$ 0.26
CAA	55.24 $\pm$ 0.71	71.58 $\pm$ 1.57	38.74 $\pm$ 0.48	31.10 $\pm$ 0.43	53.57 $\pm$ 0.82	38.45 $\pm$ 0.55	37.44 $\pm$ 1.12	37.54 $\pm$ 0.69
SADI	55.04 $\pm$ 0.39	79.05 $\pm$ 0.88	39.58 $\pm$ 0.74	31.09 $\pm$ 0.19	49.22 $\pm$ 0.63	36.02 $\pm$ 0.27	39.48 $\pm$ 0.59	39.00 $\pm$ 0.36

Table 1: Cross-lingual reasoning and generation performance across multiple benchmarks. The best score in each column is shown in bold blue , and the second-best score is shown in gray underlined . Numbers in parentheses indicate the improvement of CIF over INCLINE. For intervention-based methods, we run three random seeds and report the mean and standard deviation. Statistical significance (*) is determined using a paired t-test with p=0.05.

Method	Centroid	Drop	XCOPA	XStoryCloze	XWinograd	XCSR	XNLI	MZSRE	WMT	Flores
CIF Variant	✗	✗	53.98	81.15	46.85	38.60	58.52	40.62	42.02	39.70
CIF Variant		✗	55.81	83.11	48.62	40.00	59.29	41.08	41.86	40.08
CIF			57.36	82.76	48.73	40.19	60.32	41.14	42.70	40.92

Table 2: Ablation study of CIF averaged over four backbone models. Cross marks indicate removed CIF representations. The best score in each column is shown in bold blue , and the second-best score is shown in gray underlined . Numbers in parentheses indicate the improvement of CIF over the variant with both centroid and drop removed.

Panel of Figure 3 — Figure 3: Projection-space evidence for shared cross-lingual intervention representations. Language-specific projection perturbations are consistently aligned with the centroid, and a small number of singular vectors explain a substantial amount of projection variance.

Figure 4: Centroid-guided outlier filtering in CIF. Projection matrices removed by CIF are consistently farther from the centroid than retained matrices, indicating that the drop operation filters outlier deviations rather than discarding projections arbitrarily.

Panel of Figure 5 — Figure 5: Representation-level effect of CIF on English-Hindi FLORES examples. Before intervention, the hidden representations form separated distributions. After applying CIF, the Hindi distribution moves closer to the English distribution and even overlaps with it.

Error Type	# Examples
CIF fixes INCLINE	66
CIF fixes CIF w/o drop	59
CIF fixes both INCLINE and CIF w/o drop	3
CIF regresses vs. INCLINE	41
CIF regresses vs. CIF w/o drop	38

Table 3: Error analysis of the randomly sampled XCOPA subset.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Lang.	Premise	Choice A	Choice B	Gold	CIF w/o drop	INCLINE	CIF
et	Mehe e-postkast oli rämpsposti täis. (The man’s email inbox was full of spam.)	Ta kustutas rämpsposti. (He deleted the spam.)	Ta saatis laiali massimeili. (He sent out a mass email.)	A	A	B	A
id	Saya kosongkan kantong saya. (I emptied my pocket.)	Saya ambil sebuah potongan tiket. (I took out a ticket stub.)	Saya temukan sebuah senjata. (I found a weapon.)	A	A	B	A
sw	Msichana alitaka kumshukuru mwalimu wake wa hesabu. (The girl wanted to thank her math teacher.)	Msichana alibakia kizuizini baada ya shule. (The girl stayed after school for detention.)	Msichana alimletea tofaa mwalimu wake. (The girl brought her teacher an apple.)	B	A	B	B

Appendix

Table 4: XCOPA case study. CIF preserves correct unfiltered-fusion predictions in Estonian and Indonesian, while correcting an unfiltered-fusion error in Swahili. English translations are added for readability.

Dataset	Output	Prompt	Metric	$\mathbf{\|L\|}$
XCOPA	2-way class	Here is a premise: “{premise}”. A: “{choice1}” B: “{choice2}” What is the {question}? “A” or “B”?	F1	10
XStoryCloze	2-way class	{input} What is a possible continuation for the story given the following options? A: {ending1} B: {ending2}	F1	8
XWinograd	2-way class	{input} Replace the blank in the above sentence with the correct option: A: {option1} B: {option2}	F1	6
XCSR	multi-choice	Question: {question} {choices} Answer:	F1	15
XNLI	3-way class	Take the following as truth: {premise} Then the following statement: “{hypothesis}” is “entailment”, “neutral”, or “contradiction”?	F1	14
MZSRE	answer	{context} Question: {question} Answer:	F1	10

Appendix

Table 5: Downstream evaluation tasks used in our cross-lingual intervention experiments.

|L|

denotes the number of evaluated languages. We report F1 for classification and factual editing tasks, and BLEU for machine translation tasks.

Dataset	Language	Group	INCLINE	CIF	Gain
XCSR	hi	low-resource / distant / trimmed	29.98	42.28	+12.30 pp
XCSR	ar	distant	37.78	49.22	+11.45 pp
XCSR	vi	low-resource	39.58	51.53	+11.95 pp
XCSR	zh	distant	45.70	56.58	+10.88 pp
XCSR	sw	low-resource	35.50	39.37	+3.87 pp
XNLI	tr	distant	56.64	63.56	+6.93 pp

Appendix

Table 6: Per-language comparison between CIF and INCLINE. Scores are averaged over the four backbones after selecting a single intervention setting for each dataset–backbone pair. F1-based tasks are reported in percentages, while WMT23 and FLORES are reported in BLEU. The table includes low-resource, typologically distant, and frequently trimmed languages to examine whether CIF benefits languages beyond those retained in the fused operator.

Dataset	Held-out Lang.	Full CIF	LOO-CIF	Gap
XCOPA	ta	52.80	54.20	$+1.40$
XCOPA	vi	61.40	62.00	$+0.60$
XCOPA	zh	70.80	70.40	$-0.40$
FLORES	hi	55.77	55.14	$+0.63$
FLORES	vi	45.33	44.83	$+0.47$
FLORES	zh	38.54	37.79	$-0.75$

Appendix

Table 7: Leave-one-language-out evaluation of CIF. The held-out language is excluded when constructing the fused operator but is still used as the evaluation target. XCOPA scores are F1 percentages and FLORES scores are BLEU scores.

Dataset	Language	Linguistic Property	CIF Gain over INCLINE
XCSR	ar	Semitic / non-Latin script	$+11.45$ pp
XCSR	hi	Indo-Aryan / non-Latin script	$+12.30$ pp
XCSR	ja	Japonic / non-Latin script	$+9.58$ pp
XCSR	ur	Indo-Aryan / non-Latin script	$+7.65$ pp
XCSR	vi	Austroasiatic / low-resource setting	$+11.95$ pp
XCSR	zh	Sino-Tibetan / non-Latin script	$+10.88$ pp

Appendix

Table 8: Representative per-language gains of CIF over INCLINE for typologically distant, non-Latin-script, or lower-resource languages. Classification tasks are reported in percentage points, while FLORES is reported in BLEU. CIF improves many distant languages, suggesting that the fused operator captures transferable intervention representations beyond closely related language families. However, the regression on Chinese in FLORES shows that language-specific information can still be important for open-ended generation.

Trimming Size	XCOPA	XNLI	WMT23	FLORES
$\mathrm{drop}=0$	54.81	59.29	41.86	40.08
$\mathrm{drop}=1$	56.98	64.03	42.45	40.69
$\mathrm{drop}=2$	56.84	63.39	42.24	40.90

Appendix

Table 9: Sensitivity of CIF to the trimming size

\mathrm{drop}

. Scores are averaged over four backbones. XCOPA and XNLI are reported in F1 percentage, while WMT23 and FLORES are reported in BLEU.

Dataset	$\boldsymbol{\alpha=-1.0}$	$\boldsymbol{\alpha=-0.7}$	$\boldsymbol{\alpha=-0.4}$	$\boldsymbol{\alpha=-0.1}$	$\boldsymbol{\alpha=0.1}$	$\boldsymbol{\alpha=0.4}$	$\boldsymbol{\alpha=0.7}$	$\boldsymbol{\alpha=1.0}$
XCOPA	55.54	55.44	55.62	55.17	55.14	55.03	54.88	54.53
XNLI	52.62	52.41	53.01	57.72	56.66	53.38	53.23	53.15
WMT23	32.44	34.52	33.65	40.46	40.97	42.24	39.48	38.62
FLORES	28.46	32.56	37.79	38.57	38.00	37.57	37.64	37.90

Appendix

Table 10: Sensitivity of CIF to intervention strength

\alpha

. Each entry reports the average score under a fixed

\alpha

value across four backbones. Classification tasks are reported in F1 percentage, while WMT23 and FLORES are reported in BLEU.

Dataset	Checkpoint	Metric	Zero-shot	SADI	INCLINE	CIF
XCOPA	Qwen2.5-7B	F1	50.84	57.07	52.42	61.16
XCOPA	BLOOMZ-7B1-MT	F1	61.76	59.49	62.27	63.27
XCOPA	LLaMA-3.1-8B	F1	50.11	53.07	51.16	53.76
XCOPA	Mistral-7B	F1	49.98	50.56	50.07	51.24
XStoryCloze	Qwen2.5-7B	F1	84.96	83.16	81.42	85.01
XStoryCloze	BLOOMZ-7B1-MT	F1	77.92	78.78	80.76	83.04

Appendix

Table 11: Per-checkpoint results across all datasets. Classification tasks are reported as F1 percentage, while WMT23 and FLORES are reported as BLEU. The exact HuggingFace checkpoint names are provided in Appendix A.7 .

Dataset	Target Languages	Language Pairs
XCOPA	et, id, it, sw, ta, th, tr, vi, zh	target $\rightarrow$ en
XStoryCloze	ar, es, hi, id, ru, sw, zh	target $\rightarrow$ en
XWinograd	fr, ja, pt, ru, zh	target $\rightarrow$ en
XCSR	ar, de, es, fr, hi, it, ja, nl, pt, ru, sw, ur, vi, zh	target $\rightarrow$ en
XNLI	ar, de, el, es, fr, hi, ru, sw, th, tr, ur, vi, zh	target $\rightarrow$ en
MZSRE	de, es, fr, pt, ru, th, tr, vi, zh	target $\rightarrow$ en

Appendix

Table 12: Target languages used to construct the fused CIF centroid for each dataset. English is used as the reference language and is not counted as a target language.

Explore similar work

Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs
Jul 21, 2026Alexander ManevMultilingual Language Model Evaluation LLM Alignment
Technical University of Munich
Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer
Aug 31, 2026Minju Song, Hyeon Hwang, Junhyun Lee +1Cross-Lingual Reasoning in Language Models Low-Resource Language Processing
Korea University · Hankuk University of Foreign Studies · Noah’s Farm +1
Cross-Lingual Activation Steering for Multilingual Language Models
Jan 23, 2026Rhitabrat Pokharel, Ameeta Agrawal, Tanay NagarMultilingual Language Models Language Model Steering
Department of Computer Science, Portland State University, USA · Independent Researcher

Jul 21, 2026J/K move · Enter open · S save

Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs

Alexander Manev

Technical University of Munich

Multilingual Language Model Evaluation LLM Alignment Language Model Steering Factual Consistency Evaluation Inference-Time Language Model Alignment Causal Interventions in Language Models+2

Open paper AskPDF arXiv

Although Large Language Models (LLMs) demonstrate remarkable multilingual fluency, their internal knowledge representations remain disproportionately biased toward high-resource languages. This leads to cross-lingual factual inconsistency, where they shift their empirical answer distributions based solely on the prompt language. We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages (German, Spanish, Bulgarian), and evaluate four intervention strategies: zero-shot contextual steering (persona prompting), internal representation manipulation via Contrastive Activation Addition (CAA), and lightweight weight modification via Direct Preference Optimization (DPO) trained on benchmark-derived factual data as well as conceptual generalization data. To assess alignment, we curate a multilingual factual dataset alongside a novel generalization benchmark comprising culturally rooted queries to determine whether factual interventions transfer to broader target-centric preferences. Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization. While CAA yields sharp inconsistency benchmark shifts, it is configuration-sensitive and risks knowledge degradation. DPO-based adapters offer permanent, yet narrower and less transferable gains. These findings suggest that cross-lingual inconsistency is at least partly a selection problem, and that simple contextual interventions may outperform more invasive methods for robust, transferable alignment.

Abstract

Explore similar work

Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs

Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer

Cross-Lingual Activation Steering for Multilingual Language Models