Language steering in latent space to mitigate unintended code-switching

Andrey Goncharov; Nikolai Kondusov; Alexey Zaytsev

cs.CLOct 11, 2025

Language steering in latent space to mitigate unintended code-switching

Authors: Andrey Goncharov, Nikolai Kondusov, Alexey Zaytsev

Abstract

Multilingual Large Language Models (LLMs) often exhibit hallucinations such as unintended code-switching, reducing reliability in downstream tasks. We propose latent-space language steering, a lightweight inference-time method that identifies language directions via Principal Component Analysis (PCA) on parallel translations and steers token embeddings along these axes to control language identity. Our approach mitigates code-switching (reduced next-token distributional divergence by up to 55%, reduced Code-Switching Index by 63--99%) while preserving broad semantic direction with negligible computational overhead and requires only minimal parallel data for calibration. Additionally, we evaluate language classification using a single principal component (96-100% accuracy) and further analyze the layer-wise evolution of language representations.

Figures & tables

Figure 1: Language steering in hidden space: 1. Collect parallel translations of the same concepts; 2. LLM forward pass to get embeddings by layer; 3. Perform PCA by layer on the merged embeddings.

Panel of Figure 2 — Figure 2: Final layer language map

Panel of Figure 3 — Figure 3: Llama layer-wise language maps and explained variance

Panel of Figure 4 — Figure 4: Llama final layer language maps for each language pair

Language pair	Qwen	Llama
English - Chinese	0.98	0.98
English - Spanish	0.96	0.96
English - Russian	0.99	0.99
English - Hindi	0.98	1.00

Table 1: Last layer language classifier accuracy

Original	Unsteered	Steered
still	muy	very
going	un	surprised
not	bastante	pretty
so	tan	starting
a	sor	working
just	más	content

Table 2: Top 20 token sample (Qwen, English-Spanish)

Language pair	Qwen	Llama
English - Chinese	10.15 / 0.02 / 3.69% $\rightarrow$	9.05 / 0.05 / 4.47% $\rightarrow$
	4.52 / 0.22 / 26.46%	4.77 / 0.21 / 24.31%
English - Spanish	7.25 / 0.12 / 11.41% $\rightarrow$	6.52 / 0.12 / 11.45% $\rightarrow$
	4.90 / 0.18s / 20.23%	3.48 / 0.29 / 30.74%
English - Russian	8.70 / 0.09 / 7.41% $\rightarrow$	7.77 / 0.10 / 7.49% $\rightarrow$
	5.40 / 0.23 / 25.45%	3.49 / 0.31 / 31.96%

Table 3: KL divergence / Cosine similarity / Top-50 token overlap (no steering

\rightarrow

steering)

	Qwen	Llama
Language pair	KL-opt.	KL-opt.	CSI-opt.
English - Chinese	$-3.2$	$-5.0$	$-5.0$
English - Spanish	$-2.8$	$-3.7$	$-5.0$
English - Russian	$-2.2$	$-3.5$	$-4.0$
English - Hindi	$+5.0$	$-3.6$	$-5.0$

Table 4: Best steering strength coefficients. KL-optimized: from the distributional grid search; CSI-optimized: from the generation grid search (Section 4.4 , Llama-3.2-1B only).

		Steered
Language pair	Unsteered	KL-opt.	CSI-opt.
English - Chinese	0.64	0.03	0.03
English - Spanish	0.62	0.23	0.11
English - Russian	0.59	0.02	0.01
English - Hindi	0.81	0.04	0.01

Table 5: Code-Switching Index (CSI,

\downarrow

) of generated text (Llama-3.2-1B,

n=500

, code-switched input). English-only baseline: CSI

=0.02

en-es: “…I took on the task to enseñar desarrollo global…”
Unsteered	de que un grupo de estudiantes suecos, que habían sido enviados a la Universidad de Harvard, me escribieron para pedir ayuda…
Steered	de that I had the opportunity to work with the first two editions of the course and I was very inspired by it…
en-ru: “…About 10 years ago, I took on the task to [Russian text] …”
Degenerate steered	first year of the first year of the first year of the first year of the first year…

Table 6: Generated continuations (Llama-3.2-1B, KL-optimized). Top: steering successfully redirects Spanish to coherent English. Bottom: steering eliminates Russian but produces degenerate repetition.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 279 papers

Latent Mechanisms of Language Control in Multilingual Language Models
Aug 31, 2026Ryo Mitsuhashi, Sabri Boughorbel, Majd HawaslyMultilingual Language Models Multilingual Language Model Evaluation
Princeton University New Jersey, USA · Prince Sattam bin Abdulaziz University Riyadh, Saudi Arabia · QCRI, Hamad Bin Khalifa University, Doha, Qatar
Code-Switching Reveals Language Anchoring in Multilingual LLMs
Jun 18, 2026Jeonghyun Park, Seunghyun Yoon, Yonghyun Jun +1Multilingual Language Models Code-Switching
Chung-Ang University, Seoul, Korea · Adobe Research, USA
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
Apr 4, 2026Sing Hieng Wong, Hassan Sajjad, A. B. SiddiqueLanguage Model Steering Multilingual Language Models
Department of Computer Science, University of Kentucky, USA · Department of Computer Science, Dalhousie University, Canada

Aug 31, 2026J/K move · Enter open · S save

Latent Mechanisms of Language Control in Multilingual Language Models

Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly

Princeton University New Jersey, USA · Prince Sattam bin Abdulaziz University Riyadh, Saudi Arabia · QCRI, Hamad Bin Khalifa University, Doha, Qatar

Multilingual Language Models Multilingual Language Model Evaluation Code-Switching LLM Interpretability Causal Interventions in Language Models

Open paper AskPDF arXiv

Multilingual large language models can exhibit unintended code-switching -- unnecessarily alternating between languages during generation. We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders: activation value-based selection (ValSel), activation frequency-based selection (FreqSel), and LLM-generated latent annotation-based selection (AnnSel). To evaluate the efficacy of these methods in identifying language-controlling latents, we introduce two multilingual benchmarks that exhibit code-switching for fine-grained analysis of language steering across seven languages. Through targeted intervention experiments on Gemma-2-2B and Qwen3-4B, we find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance, while AnnSel offering interpretable latent selection through explicit language annotations. A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction. Code and data can be found at https://github.com/rm-3284/Latent-Mechanism-Multilingual.

Abstract

Language Model Steering

Explore similar work

Latent Mechanisms of Language Control in Multilingual Language Models

Code-Switching Reveals Language Anchoring in Multilingual LLMs

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering