Language Steering for Multilingual In-Context Learning

Neeraja Kirtane; Kuan-Hao Huang

cs.CLFeb 2, 2026

Language Steering for Multilingual In-Context Learning

Authors: Neeraja Kirtane, Kuan-Hao Huang

Organizations: Texas A&M University

Abstract

If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions. To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge. We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states. Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones. Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.

Figures & tables

Figure 1: Overview of language steering for multilingual in-context learning. (a) Offline vector computation: parallel English (

L_{s}

) and target-language (

L_{t}

) question-answer sequences are passed through the LLM, and their hidden states at layer

t

are mean-pooled. The language vector

\mathbf{v}(t)

is the mean of the paired differences across

N

samples. It is computed once per language, with no parameter updates. (b) Inference-time steering: given English demonstrations and a target-language question, we add

\alpha\ \mathbf{v}(t)

to the hidden states of the demonstration tokens at layer

t

, leaving the question tokens unchanged (more configs detailed in Section 2.3 ). (c) Prompt conditions: the baseline pairs English demonstrations with a target-language question, the oracle uses target-language demonstrations, and our method matches the two languages in activation space.

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	62.57	60.78	62.87	63.77	–	–	–	–
Basque	32.14	35.71	36.90	52.38	–	–	–	–	–	–	–	–
Bengali	57.14	55.95	61.90	58.33	–	–	–	–	57.49	62.87	59.58	61.38
Bulgarian	–	–	–	–	56.29	61.98	61.68	66.17	–	–	–	–
Catalan	64.29	64.29	69.05	76.19	–	–	–	–	–	–	–	–

Table 1: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP for the Llama-3.1-8B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language excluding the oracle.

Model	MGSM				XNLI				MSVAMP
Model	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Llama-3.1-8B-Instruct	61.01	63.19	65.87	68.16	61.38	61.68	64.31	65.04	65.84	68.66	68.60	70.06
Qwen2.5-7B-Instruct	68.95	69.94	71.83	73.21	74.81	73.27	75.51	74.53	75.18	76.94	76.68	77.65
Qwen2.5-14B-Instruct	81.55	82.84	84.23	84.72	72.73	73.91	74.66	76.45	82.10	82.90	83.90	83.03

Table 2: Average performance comparison across three datasets and three models. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Detailed per-language results are in Appendix A .

	B	Ours	Nearest	Farthest	Random
eu	32.14	36.90	29.76 sw	27.38 zh	33.73
bn	57.14	61.90	58.33 th	59.52 zh	59.92
ca	64.29	69.05	64.29 gl	64.29 zh	63.89
zh	67.86	71.43	69.05 ja	69.05 bn	67.46
fr	61.90	70.24	65.48 es	65.48 bn	64.29
gl	64.29	73.81	71.43 ca	67.86 zh	65.48

Table 3: Steering with another language’s vector on MGSM (Llama-3.1-8B-Instruct). Ours = own vector; Nearest/Farthest = vector of the closest/most distant language by layer-10 cosine distance (source language in small type); Random = Gaussian, mean of 3 draws. All vectors rescaled to the target’s own norm. Bold : best per language.

Vector $\rightarrow$ Eval	B	MFS	Ours	CT	OR
MGSM $\rightarrow$ XNLI	75.19	74.22	75.92	75.96	75.58
XNLI $\rightarrow$ MGSM	73.64	74.32	76.53	77.04	77.55
MGSM $\rightarrow$ MSVAMP	75.18	76.94	76.68	75.18	77.65
MSVAMP $\rightarrow$ MGSM	71.69	73.54	74.34	72.22	75.79
MSVAMP $\rightarrow$ XNLI	75.19	74.25	75.92	74.04	75.58
XNLI $\rightarrow$ MSVAMP	76.05	77.41	77.63	75.83	77.50

Table 4: Average cross-task transfer accuracy across languages on Qwen2.5-7B-Instruct. Steering vectors are computed on one task and evaluated on another. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , CT = Cross-Transfer, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best non-oracle score.

Figure 2: Hierarchical clustering of steering vectors at layer 10 of Llama-3.1-8B-Instruct using the FLORES dataset. Edge labels show cosine distances (lower = more similar); colours indicate language families.

Lang.	Baseline	Ours	ST	$\Delta$ Tok.	Lat./Q. (s)
zh	67.86	71.43	71.43	227	12.64
bn	57.14	61.90	57.14	872	26.87
de	66.67	75.00	72.62	231	13.40
es	77.38	76.19	80.95	244	12.25
fr	61.90	70.24	67.86	264	15.14
ja	55.95	55.95	57.14	310	16.97

Table 5: Accuracy (%) of baseline, steering, and self-translate (few-shot demos re-translated per query), with self-translate’s extra generated tokens (

\Delta

Tok.) and latency per query. Steering adds no extra tokens.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	70.96	71.26	71.56	74.55	–	–	–	–
Basque	21.43	15.48	26.19	26.19	–	–	–	–	–	–	–	–
Bengali	58.33	66.67	58.33	65.48	–	–	–	–	65.57	70.96	67.07	76.35
Bulgarian	–	–	–	–	79.34	76.65	80.24	78.74	–	–	–	–
Catalan	80.95	78.57	84.52	85.71	–	–	–	–	–	–	–	–

Appendix

Table 6: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP on Qwen2.5-7B-Instruct. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language (excluding Oracle).

Language	MGSM				XNLI				MSVAMP
Language	B	MFS	Ours	OR	B	MFS	Ours	OR	B	MFS	Ours	OR
Arabic	–	–	–	–	73.35	75.15	74.85	78.44	–	–	–	–
Basque	47.62	54.76	47.62	59.52	–	–	–	–	–	–	–	–
Bengali	76.19	80.95	76.19	77.38	–	–	–	–	75.45	76.95	76.95	76.05
Bulgarian	–	–	–	–	76.35	78.14	77.84	79.64	–	–	–	–
Catalan	84.52	89.29	89.29	94.05	–	–	–	–	–	–	–	–

Appendix

Table 7: Detailed per-language accuracy results on MGSM, XNLI, and MSVAMP for the Qwen2.5-14B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language.

Dataset	Model	Performance
Dataset	Model	R	Ours
XNLI	Llama-3.1-8B-Instruct	63.37	64.31
	Qwen2.5-7B-Instruct	75.32	75.51
	Qwen2.5-14B-Instruct	73.37	74.66

Appendix

Table 8: Comparison of random steering vs. our method across models. R = Random Steering.

Language	Base	Mean	Last
Bengali	57.14	61.90	53.57
Catalan	64.29	69.05	61.90
German	66.67	75.00	65.48
Spanish	77.38	76.19	71.43
Basque	32.14	36.90	32.14
French	61.90	70.24	60.71

Appendix

Table 9: Accuracy using mean-pooling vs. last-token hidden states (Llama-3.1-8B-Instruct, MGSM). Mean = Mean, Last = Last Token.

Language	B	OF	AF	OQ	ENT
Bengali	57.14	60.71	61.90	60.71	55.95
Catalan	64.29	66.67	69.05	67.86	69.05
German	66.67	75.00	66.67 †	69.05	64.29
Spanish	77.38	73.81	75.00	76.19	72.62
Basque	32.14	35.71	36.90	33.33	36.90
French	61.90	67.86	64.29	67.86	70.24

Appendix

Table 10: Per-language accuracy across steering position configurations for Llama-3.1-8B-Instruct on MGSM. B = Baseline, OF = On Few-shot, AF = After Few-shot, OQ = On Question, ENT = Entire Prompt. † No configuration improved over the baseline on validation, so the baseline score is reported. Bold indicates the best score per language.

Language	MGSM (vector) $\rightarrow$ XNLI (eval)					XNLI (vector) $\rightarrow$ MGSM (eval)
Language	B	MFS	Ours	CT	OR	B	MFS	Ours	CT	OR
Chinese	78.14	77.84	77.84	78.14	79.34	84.52	88.10	86.90	90.48	79.76
Thai	70.96	73.05	70.06	70.06	72.46	79.76	82.14	79.76	82.14	85.71
Swahili	53.29	47.31	55.09	54.79	54.79	19.05	16.67	26.19	25.00	33.33
Russian	78.14	79.04	79.64	79.94	78.74	85.71	85.71	88.10	88.10	85.71
French	81.44	82.34	82.34	82.63	82.63	78.57	80.95	79.76	79.76	84.52

Appendix

Table 11: Cross-task transfer results across all task-vector combinations between MGSM, MSVAMP, and XNLI on Qwen2.5-7B-Instruct. For each task pair, we compute steering vectors from one task and evaluate on another task. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , CT = Cross-Transfer, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score.

Language	MGSM					XNLI					MSVAMP
Language	B	FLORES	Single	Ours	OR	B	FLORES	Single	Ours	OR	B	FLORES	Single	Ours	OR
Arabic	–	–	–	–	–	62.57	61.08	63.17	62.87	63.77	–	–	–	–	–
Basque	32.14	27.38	36.90	36.90	52.38	–	–	–	–	–	–	–	–	–	–
Bengali	57.14	57.14	60.71	61.90	58.33	–	–	–	–	–	57.49	57.49	59.28	59.58	61.38
Bulgarian	–	–	–	–	–	56.29	58.68	60.18	61.68	66.17	–	–	–	–	–
Catalan	64.29	65.48	66.67	69.05	76.19	–	–	–	–	–	–	–	–	–	–

Appendix

Table 12: Alternative vector configurations per language. B = Baseline, FLORES = vectors computed from general-purpose parallel sentences ( Costa-Jussà et al., 2022 ) , Single = our method with single question-answer pairs instead of concatenated few-shot sequences, Ours = our full few-shot method, OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates best among B, FLORES, Single, and Ours.

Language	Baseline	Ours	Other-data
Chinese	67.9	71.4	69.0
Thai	57.1	61.9	58.3
Bengali	57.1	61.9	56.0
German	66.7	75.0	76.2
Spanish	77.4	76.2	75.0
French	61.9	70.2	66.7

Appendix

Table 13: MGSM accuracy on Llama-3.1-8B-Instruct by using data for vector computation and validation from a different math dataset (MathOctopus) and testing on MGSM. We use the subset of 9 languages common to both the datasets.

	Total	Compute	Val	Test
MGSM	250	83	83	84
XNLI	1,000	333	333	334
MSVAMP	1,000	333	333	334

Appendix

Table 14: Examples per language in each split.

Language	MGSM	MSVAMP	XNLI
Arabic	—	—	10/3.0
Basque	5/1.0	—	—
Bengali	20/1.0	5/0.5	—
Bulgarian	—	—	10/3.0
Catalan	20/3.0	—	—
Chinese	30/0.5	5/0.5	5/3.0

Appendix

Table 15: Best layer/

\alpha

per language and task for Llama-3.1-8B-Instruct. Each cell shows layer /

\alpha

; — indicates the language was not evaluated on that task.

Language	MGSM					XNLI					MSVAMP
Language	B	MFS	Per-Lang	Task-Lvl	OR	B	MFS	Per-Lang	Task-Lvl	OR	B	MFS	Per-Lang	Task-Lvl	OR
Arabic	–	–	–	–	–	62.57	60.78	62.87	61.38	63.77	–	–	–	–	–
Basque	32.14	35.70	36.90	32.14	52.38	–	–	–	–	–	–	–	–	–	–
Bengali	57.14	55.95	61.90	53.57	58.33	–	–	–	–	–	57.49	62.87	59.58	59.58	61.38
Bulgarian	–	–	–	–	–	56.29	61.98	61.68	60.18	66.17	–	–	–	–	–
Catalan	64.29	64.28	69.05	63.10	76.19	–	–	–	–	–	–	–	–	–	–

Appendix

Table 16: Per-language accuracy under per-language and task-level hyperparameter selection on MGSM, XNLI, and MSVAMP for the Llama-3.1-8B-Instruct model. Gray dashes indicate that the dataset does not contain that language. B = Baseline, MFS = Multilingual Few-Shot Baseline Tu et al. (2025) , OR = Oracle (target-language few-shot reference, not intended for direct comparison). Bold indicates the best score per language excluding the oracle.

Lang.	Baseline	Ours	Oracle
bn	$56.35\pm 2.48$	$62.70\pm 1.37$	$58.33\pm 3.57$
ca	$61.90\pm 2.06$	$69.84\pm 1.37$	$75.40\pm 1.37$
de	$69.05\pm 3.15$	$74.21\pm 1.37$	$73.81\pm 2.38$
es	$73.02\pm 3.83$	$76.98\pm 1.37$	$76.59\pm 1.82$
eu	$31.35\pm 3.64$	$37.30\pm 3.00$	$51.98\pm 0.69$
fr	$66.27\pm 4.18$	$71.03\pm 2.48$	$69.44\pm 4.18$

Appendix

Table 17: Performance across languages over multiple random seeds for the MGSM dataset on the Llama-3.1-8b-Instruct model. Values are mean

\pm

std.

Figure 3: Hierarchical clustering of language vectors computed from task data at layer 10 of Llama-3.1-8B-Instruct. Each language’s vector is computed from its available dataset: MGSM for Basque, Catalan, and Galician; XNLI for Arabic, Bulgarian, Greek, Hindi, Turkish, Urdu, and Vietnamese; Edge labels show cosine distances (lower = more similar); colours indicate language families. The closest pairs (Catalan–Galician, Hindi–Urdu, Bulgarian–Greek) match those in the FLORES-based clustering (Figure 2 ), while coarser structure partly reflects the source corpus.

Lang	Baseline PPL	Steered PPL	Ratio
bn	1.123	1.126	1.002
ca	1.260	1.260	1.000
de	1.238	1.250	1.009
es	1.231	1.233	1.002
eu	1.320	1.362	1.031
fr	1.235	1.273	1.031

Appendix

Table 18: Perplexity of steered vs. baseline generations, scored under an unsteered forward pass (teacher-forced NLL, response tokens only). A ratio near 1.0 indicates steering does not push generations off the model’s natural distribution. Computed for MGSM data on Llama-3.1-8B-Instruct.

Explore similar work

Jan 23, 2026

Cross-Lingual Activation Steering for Multilingual Language Models

Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages. Prior work attributes this gap to imbalances between shared and language-specific neurons in multilingual representations. We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates neuron activations. We evaluate CLAS on classification and generation benchmarks, achieving average improvements of 2.3% (Acc.) and 3.4% (F1) respectively, while maintaining high-resource language performance. We discover that effective transfer operates through functional divergence rather than strict alignment; performance gains correlate with increased language cluster separation. Our results demonstrate that targeted activation steering can unlock latent multilingual capacity in existing models without modification to model weights.

Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar

Department of Computer Science, Portland State University, USA · Independent Researcher

Multilingual Language Models Language Model Steering Causal Interventions in Language Models Cross-Lingual Transfer

Ask PDF

Mar 3, 2025

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

Instruction following is a critical ability for Large Language Models to be used directly by humans. This often requires supervised fine-tuning on curated instruction datasets, sometimes complemented with an alignment step. However, in multilingual scenarios, obtaining high-quality data for these stages remains challenging, motivating the exploration of In-Context Learning (ICL) as a possible alternative. In this work, we study whether ICL can serve as a substitute for Instruction Tuning in multilingual language models, while also examining how the comparison changes with model scale. Our results indicate that a gap remains between ICL and Instruction Tuning, motivating further research to reduce it.

David Ponce, Thierry Etchegoyhen

Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) · University of the Basque Country - EHU

Multilingual Language Models Multilingual Language Model Evaluation In-Context Learning Instruction Tuning

Ask PDF

Jun 16, 2026

When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning

Cross-lingual transfer in multilingual NLP has been widely explored in supervised fine-tuning contexts, where factors like data availability and linguistic similarity largely determine transfer quality. As the field shifts toward few-shot In-Context Learning (ICL), it is often presumed that insights from fine-tuning carry over unchanged. Yet this assumption has not been rigorously evaluated, leaving open the question of how to choose source languages for cross-lingual ICL. We conduct a broad empirical study of cross-lingual transfer in ICL spanning seven tasks, six models, and a typologically diverse set of languages. We further analyze language confusion, a key obstacle for generative tasks in cross-lingual ICL. Our results show that conventional fine-tuning-based expectations do not consistently apply in the ICL regime and point to alternative heuristics for selecting source languages effectively.

Fred Philippy, Siwen Guo, Jacques Klein +1

Snt, University of Luxembourg, Luxembourg · Luxembourg Institute of Science and Technology, Luxembourg

Multilingual Language Model Evaluation In-Context Learning Cross-Lingual Transfer

Ask PDF