Language Steering for Multilingual In-Context Learning

Neeraja Kirtane; Kuan-Hao Huang

cs.CLFeb 2, 2026

Language Steering for Multilingual In-Context Learning

Authors: Neeraja Kirtane, Kuan-Hao Huang

Organizations: Texas A&M University

Abstract

If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions. To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge. We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states. Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones. Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.

Figures & tables

Figure 1: Overview of language steering for multilingual in-context learning. (a) Offline vector computation: parallel English (

L_{s}

) and target-language (

L_{t}

) question-answer sequences are passed through the LLM, and their hidden states at layer

t

are mean-pooled. The language vector

\mathbf{v}(t)

is the mean of the paired differences across

N

samples. It is computed once per language, with no parameter updates. (b) Inference-time steering: given English demonstrations and a target-language question, we add

\alpha\ \mathbf{v}(t)

to the hidden states of the demonstration tokens at layer

t

, leaving the question tokens unchanged (more configs detailed in Section 2.3 ). (c) Prompt conditions: the baseline pairs English demonstrations with a target-language question, the oracle uses target-language demonstrations, and our method matches the two languages in activation space.

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	62.57	60.78	62.87	63.77	–	–	–	–
Basque	32.14	35.71	36.90	52.38	–	–	–	–	–	–	–	–
Bengali	57.14	55.95	61.90	58.33	–	–	–	–	57.49	62.87	59.58	61.38
Bulgarian	–	–	–	–	56.29	61.98	61.68	66.17	–	–	–	–
Catalan	64.29	64.29	69.05	76.19	–	–	–	–	–	–	–	–

Table 1: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP for the Llama-3.1-8B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language excluding the oracle.

Model	MGSM				XNLI				MSVAMP
Model	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Llama-3.1-8B-Instruct	61.01	63.19	65.87	68.16	61.38	61.68	64.31	65.04	65.84	68.66	68.60	70.06
Qwen2.5-7B-Instruct	68.95	69.94	71.83	73.21	74.81	73.27	75.51	74.53	75.18	76.94	76.68	77.65
Qwen2.5-14B-Instruct	81.55	82.84	84.23	84.72	72.73	73.91	74.66	76.45	82.10	82.90	83.90	83.03

Table 2: Average performance comparison across three datasets and three models. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Detailed per-language results are in Appendix A .

	B	Ours	Nearest	Farthest	Random
eu	32.14	36.90	29.76 sw	27.38 zh	33.73
bn	57.14	61.90	58.33 th	59.52 zh	59.92
ca	64.29	69.05	64.29 gl	64.29 zh	63.89
zh	67.86	71.43	69.05 ja	69.05 bn	67.46
fr	61.90	70.24	65.48 es	65.48 bn	64.29
gl	64.29	73.81	71.43 ca	67.86 zh	65.48

Table 3: Steering with another language’s vector on MGSM (Llama-3.1-8B-Instruct). Ours = own vector; Nearest/Farthest = vector of the closest/most distant language by layer-10 cosine distance (source language in small type); Random = Gaussian, mean of 3 draws. All vectors rescaled to the target’s own norm. Bold : best per language.

Vector $\rightarrow$ Eval	B	MFS	Ours	CT	OR
MGSM $\rightarrow$ XNLI	75.19	74.22	75.92	75.96	75.58
XNLI $\rightarrow$ MGSM	73.64	74.32	76.53	77.04	77.55
MGSM $\rightarrow$ MSVAMP	75.18	76.94	76.68	75.18	77.65
MSVAMP $\rightarrow$ MGSM	71.69	73.54	74.34	72.22	75.79
MSVAMP $\rightarrow$ XNLI	75.19	74.25	75.92	74.04	75.58
XNLI $\rightarrow$ MSVAMP	76.05	77.41	77.63	75.83	77.50

Table 4: Average cross-task transfer accuracy across languages on Qwen2.5-7B-Instruct. Steering vectors are computed on one task and evaluated on another. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , CT = Cross-Transfer, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best non-oracle score.

Figure 2: Hierarchical clustering of steering vectors at layer 10 of Llama-3.1-8B-Instruct using the FLORES dataset. Edge labels show cosine distances (lower = more similar); colours indicate language families.

Lang.	Baseline	Ours	ST	$\Delta$ Tok.	Lat./Q. (s)
zh	67.86	71.43	71.43	227	12.64
bn	57.14	61.90	57.14	872	26.87
de	66.67	75.00	72.62	231	13.40
es	77.38	76.19	80.95	244	12.25
fr	61.90	70.24	67.86	264	15.14
ja	55.95	55.95	57.14	310	16.97

Table 5: Accuracy (%) of baseline, steering, and self-translate (few-shot demos re-translated per query), with self-translate’s extra generated tokens (

\Delta

Tok.) and latency per query. Steering adds no extra tokens.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	70.96	71.26	71.56	74.55	–	–	–	–
Basque	21.43	15.48	26.19	26.19	–	–	–	–	–	–	–	–
Bengali	58.33	66.67	58.33	65.48	–	–	–	–	65.57	70.96	67.07	76.35
Bulgarian	–	–	–	–	79.34	76.65	80.24	78.74	–	–	–	–
Catalan	80.95	78.57	84.52	85.71	–	–	–	–	–	–	–	–

Appendix

Table 6: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP on Qwen2.5-7B-Instruct. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language (excluding Oracle).

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	73.35	75.15	74.85	78.44	–	–	–	–
Basque	47.62	54.76	47.62	59.52	–	–	–	–	–	–	–	–
Bengali	76.19	80.95	76.19	77.38	–	–	–	–	75.45	76.95	76.95	76.05
Bulgarian	–	–	–	–	76.35	78.14	77.84	79.64	–	–	–	–
Catalan	84.52	89.29	89.29	94.05	–	–	–	–	–	–	–	–

Appendix

Table 7: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP for the Qwen2.5-14B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language.

Dataset	Model	Performance
Dataset	Model	R	Ours
XNLI	Llama-3.1-8B-Instruct	63.37	64.31
	Qwen2.5-7B-Instruct	75.32	75.51
	Qwen2.5-14B-Instruct	73.37	74.66

Appendix

Table 8: Comparison of random steering vs. our method across models. R = Random Steering.

Language	Base	Mean	Last
Bengali	57.14	61.90	53.57
Catalan	64.29	69.05	61.90
German	66.67	75.00	65.48
Spanish	77.38	76.19	71.43
Basque	32.14	36.90	32.14
French	61.90	70.24	60.71

Appendix

Table 9: Accuracy using mean-pooling vs. last-token hidden states (Llama-3.1-8B-Instruct, MGSM). Mean = Mean, Last = Last Token.

Language	B	OF	AF	OQ	ENT
Bengali	57.14	60.71	61.90	60.71	55.95
Catalan	64.29	66.67	69.05	67.86	69.05
German	66.67	75.00	66.67 †	69.05	64.29
Spanish	77.38	73.81	75.00	76.19	72.62
Basque	32.14	35.71	36.90	33.33	36.90
French	61.90	67.86	64.29	67.86	70.24

Appendix

Table 10: Per-language accuracy across steering position configurations for Llama-3.1-8B-Instruct on MGSM. B = Baseline, OF = On Few-shot, AF = After Few-shot, OQ = On Question, ENT = Entire Prompt. † No configuration improved over the baseline on validation, so the baseline score is reported. Bold indicates the best score per language.

Language	MGSM (vector) $\rightarrow$ XNLI (eval)					XNLI (vector) $\rightarrow$ MGSM (eval)
Language	B	MFS	Ours	CT	OR	B	MFS	Ours	CT	OR
Chinese	78.14	77.84	77.84	78.14	79.34	84.52	88.10	86.90	90.48	79.76
Thai	70.96	73.05	70.06	70.06	72.46	79.76	82.14	79.76	82.14	85.71
Swahili	53.29	47.31	55.09	54.79	54.79	19.05	16.67	26.19	25.00	33.33
Russian	78.14	79.04	79.64	79.94	78.74	85.71	85.71	88.10	88.10	85.71
French	81.44	82.34	82.34	82.63	82.63	78.57	80.95	79.76	79.76	84.52

Appendix

Table 11: Cross-task transfer results across all task-vector combinations between MGSM, MSVAMP, and XNLI on Qwen2.5-7B-Instruct. For each task pair, we compute steering vectors from one task and evaluate on another task. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , CT = Cross-Transfer, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score.

Language	MGSM					XNLI					MSVAMP
Language	B	FLORES	Single	Ours	OR	B	FLORES	Single	Ours	OR	B	FLORES	Single	Ours	OR
Arabic	–	–	–	–	–	62.57	61.08	63.17	62.87	63.77	–	–	–	–	–
Basque	32.14	27.38	36.90	36.90	52.38	–	–	–	–	–	–	–	–	–	–
Bengali	57.14	57.14	60.71	61.90	58.33	–	–	–	–	–	57.49	57.49	59.28	59.58	61.38
Bulgarian	–	–	–	–	–	56.29	58.68	60.18	61.68	66.17	–	–	–	–	–
Catalan	64.29	65.48	66.67	69.05	76.19	–	–	–	–	–	–	–	–	–	–

Appendix

Table 12: Alternative vector configurations per language. B = Baseline, FLORES = vectors computed from general-purpose parallel sentences ( Costa-Jussà et al., 2022 ) , Single = our method with single question-answer pairs instead of concatenated few-shot sequences, Ours = our full few-shot method, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates best among B, FLORES, Single, and Ours.

Language	Baseline	Ours	Other-data
Chinese	67.9	71.4	69.0
Thai	57.1	61.9	58.3
Bengali	57.1	61.9	56.0
German	66.7	75.0	76.2
Spanish	77.4	76.2	75.0
French	61.9	70.2	66.7

Appendix

Table 13: MGSM accuracy on Llama-3.1-8B-Instruct by using data for vector computation and validation from a different math dataset (MathOctopus) and testing on MGSM. We use the subset of 9 languages common to both the datasets.

	Total	Compute	Val	Test
MGSM	250	83	83	84
XNLI	1,000	333	333	334
MSVAMP	1,000	333	333	334

Appendix

Table 14: Examples per language in each split.

Language	MGSM	MSVAMP	XNLI
Arabic	—	—	10/3.0
Basque	5/1.0	—	—
Bengali	20/1.0	5/0.5	—
Bulgarian	—	—	10/3.0
Catalan	20/3.0	—	—
Chinese	30/0.5	5/0.5	5/3.0

Appendix

Table 15: Best layer/

\alpha

per language and task for Llama-3.1-8B-Instruct. Each cell shows layer /

\alpha

; — indicates the language was not evaluated on that task.

Language	MGSM					XNLI					MSVAMP
Language	B	MFS	Per-Lang	Task-Lvl	OR	B	MFS	Per-Lang	Task-Lvl	OR	B	MFS	Per-Lang	Task-Lvl	OR
Arabic	–	–	–	–	–	62.57	60.78	62.87	61.38	63.77	–	–	–	–	–
Basque	32.14	35.70	36.90	32.14	52.38	–	–	–	–	–	–	–	–	–	–
Bengali	57.14	55.95	61.90	53.57	58.33	–	–	–	–	–	57.49	62.87	59.58	59.58	61.38
Bulgarian	–	–	–	–	–	56.29	61.98	61.68	60.18	66.17	–	–	–	–	–
Catalan	64.29	64.28	69.05	63.10	76.19	–	–	–	–	–	–	–	–	–	–

Appendix

Table 16: Per-language accuracy under per-language and task-level hyperparameter selection on MGSM, XNLI, and MSVAMP for the Llama-3.1-8B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language excluding the oracle.

Lang.	Baseline	Ours	Oracle
bn	$56.35\pm 2.48$	$62.70\pm 1.37$	$58.33\pm 3.57$
ca	$61.90\pm 2.06$	$69.84\pm 1.37$	$75.40\pm 1.37$
de	$69.05\pm 3.15$	$74.21\pm 1.37$	$73.81\pm 2.38$
es	$73.02\pm 3.83$	$76.98\pm 1.37$	$76.59\pm 1.82$
eu	$31.35\pm 3.64$	$37.30\pm 3.00$	$51.98\pm 0.69$
fr	$66.27\pm 4.18$	$71.03\pm 2.48$	$69.44\pm 4.18$

Appendix

Table 17: Performance across languages over multiple random seeds for the MGSM dataset on the Llama-3.1-8b-Instruct model. Values are mean

\pm

std.

Figure 3: Hierarchical clustering of language vectors computed from task data at layer 10 of Llama-3.1-8B-Instruct. Each language’s vector is computed from its available dataset: MGSM for Basque, Catalan, and Galician; XNLI for Arabic, Bulgarian, Greek, Hindi, Turkish, Urdu, and Vietnamese; Edge labels show cosine distances (lower = more similar); colours indicate language families. The closest pairs (Catalan–Galician, Hindi–Urdu, Bulgarian–Greek) match those in the FLORES-based clustering (Figure 2 ), while coarser structure partly reflects the source corpus.

Lang	Baseline PPL	Steered PPL	Ratio
bn	1.123	1.126	1.002
ca	1.260	1.260	1.000
de	1.238	1.250	1.009
es	1.231	1.233	1.002
eu	1.320	1.362	1.031
fr	1.235	1.273	1.031

Appendix

Table 18: Perplexity of steered vs. baseline generations, scored under an unsteered forward pass (teacher-forced NLL, response tokens only). A ratio near 1.0 indicates steering does not push generations off the model’s natural distribution. Computed for MGSM data on Llama-3.1-8B-Instruct.

Explore similar work

Cross-Lingual Activation Steering for Multilingual Language Models
Jan 23, 2026Rhitabrat Pokharel, Ameeta Agrawal, Tanay NagarMultilingual Language Models Language Model Steering
Department of Computer Science, Portland State University, USA · Independent Researcher
In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models
Mar 3, 2025David Ponce, Thierry EtchegoyhenMultilingual Language Models Multilingual Language Model Evaluation
Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) · University of the Basque Country - EHU
When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning
Jun 16, 2026Fred Philippy, Siwen Guo, Jacques Klein +1Multilingual Language Model Evaluation In-Context Learning
Snt, University of Luxembourg, Luxembourg · Luxembourg Institute of Science and Technology, Luxembourg

Jan 23, 2026J/K move · Enter open · S save

Cross-Lingual Activation Steering for Multilingual Language Models

Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar

Department of Computer Science, Portland State University, USA · Independent Researcher

Multilingual Language Models Language Model Steering Causal Interventions in Language Models Cross-Lingual Transfer

Open paper AskPDF arXiv

Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages. Prior work attributes this gap to imbalances between shared and language-specific neurons in multilingual representations. We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates neuron activations. We evaluate CLAS on classification and generation benchmarks, achieving average improvements of 2.3% (Acc.) and 3.4% (F1) respectively, while maintaining high-resource language performance. We discover that effective transfer operates through functional divergence rather than strict alignment; performance gains correlate with increased language cluster separation. Our results demonstrate that targeted activation steering can unlock latent multilingual capacity in existing models without modification to model weights.

Abstract

Explore similar work

Cross-Lingual Activation Steering for Multilingual Language Models

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning