LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

Sing Hieng Wong; Hassan Sajjad; A. B. Siddique

cs.CLApr 4, 2026

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

Authors: Sing Hieng Wong, Hassan Sajjad, A. B. Siddique

Organizations: Department of Computer Science, University of Kentucky, USA · Department of Computer Science, Dalhousie University, Canada

Abstract

Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult. Representation-level steering addresses this by adding language-specific vectors to model activations at inference time, but identifying language-specific directions in the residual stream often relies on multilingual or parallel data that can be expensive to obtain. Sparse autoencoders (SAEs) decompose residual activations into interpretable, sparse feature directions and offer a natural basis for this search, yet existing SAE-based approaches face the same data constraint. We introduce LangFIR (Language Feature Identification via Random-token Filtering), a method that discovers language-specific SAE features using only a small amount of monolingual data and random-token sequences. Many SAE features consistently activated by target-language inputs do not encode language identity. Random-token sequences surface these language-agnostic features, allowing LangFIR to filter them out and isolate a sparse set of language-specific features. We show that these features are extremely sparse, highly selective for their target language, and causally important: directional ablation increases cross-entropy loss only for the corresponding language. Using these features to construct steering vectors for the multilingual generation control task, LangFIR achieves the best average accuracy x BLEU among steering methods across three models (Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B), three datasets, and twelve target languages, outperforming the strongest monolingual baseline by up to 4.7x and surpassing methods that use parallel data. Our results suggest that language identity in multilingual LLMs is localized in a sparse set of feature directions discoverable with monolingual data. Code is available at https://github.com/JaMussCraft/LangFIR

Figures & tables

Figure 1: Overview of LangFIR. (1) For each target-language sentence, a random-token sequence is generated by sampling from the tokenizer’s vocabulary. (2) Residual-stream activations are extracted at a layer $l$ and encoded through SAE. (3) Sample-wise filtering is performed with activation frequency threshold $\tau$ to identify features that activate frequently for target-language sentences ( language-consistent ) and for random-token sequences ( language-agnostic ). (4) Removing language-agnostic features from language-consistent features yields a sparse set of language-specific features .

Panel of Figure 2 — Figure 2: Analyses of language-specific feature identification across sample sizes on Llama 3.1 8B. (a) Overlap between language-consistent and language-agnostic feature sets approaches 100%. (b) Fewer than 5 language-specific features remain per language. Both quantities stabilize by ${\sim}$ 100 sample size.

Panel of Figure 3 — Figure 3: Analyses of language-specific feature properties on Llama 3.1 8B. (a) Rank-1 features dominate in mean activation magnitude. (b) Language-specific features activate selectively on their target language and remain near zero on others.

Panel of Figure 4 — Figure 4: Change in cross-entropy after ablating the top-2 language-specific features per layer on Llama 3.1 8B. Ablation increases loss selectively for the target language. English is a notable exception.

	Non-steering		Steering
Model	No Steering	Prompt Only	PCA	LDA	Zhong (M)	Zhong (P)	SAE-DM	Gated-DM	DiffMean	LangFIR
Gemma 3 1B	0.8	22.2	1.6	2.0	3.3	9.5	14.7	9.0	15.3	15.5
Gemma 3 4B	2.6	35.5	3.3	15.2	10.5	13.9	23.2	10.8	23.9	26.6
Llama 3.1 8B	1.5	34.7	2.9	22.7	6.6	7.6	4.7	22.6	22.2	23.4
Avg	1.6	30.8	2.6	13.3	6.8	10.3	14.2	14.1	20.5	21.8

Table 1: ACC

\times

BLEU averaged over twelve target languages and three datasets (FLORES+, WikiMatrix, Tatoeba). The best result among steering methods within each model is bolded .

Panel of Figure 5 — Figure 5: Ablation of number of top- $k$ language-specific features, sample size, and feature-overlap removal on Llama 3.1 8B. (a) Even $k{=}1$ achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics dramatically.

	Non-steering		Steering
Model	No Steering	Prompt Only	PCA	LDA	Zhong (M)	Zhong (P)	SAE-DM	Gated-DM	DiffMean	LangFIR
Gemma 3 1B	12.7	12.3	10.6	12.7	11.1	15.1	13.3	13.6	14.4	16.7
Gemma 3 4B	14.6	7.9	15.6	14.7	11.3	13.3	22.5	19.3	21.7	20.0
Llama 3.1 8B	12.3	12.2	11.6	11.8	10.6	11.2	11.1	20.8	20.7	21.6
Avg	13.2	10.8	12.6	13.0	11.0	13.2	15.6	17.9	18.9	19.4

Table 2: Cross-lingual answer generation results: F1 averaged over five target languages (es, de, vi, tr, zh). The best result among steering methods within each model is bolded .

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Model ID	SAE Release	SAE ID
google/gemma-3-1b-pt	gemma-scope-2-1b-pt-res-all	layer_{layer}_width_16k_l0_small
google/gemma-3-4b-pt	gemma-scope-2-4b-pt-res-all	layer_{layer}_width_16k_l0_small
meta-llama/Llama-3.1-8B	llama_scope_lxr_8x	l{layer}r_8x

Appendix

Table 3: Models and SAEs used in our experiments.

Hyperparameter	Values
Intervention layer percentile	0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Steering scalar $\alpha$ (Zhong Mono/Para)	0.2, 0.6, 1.0, 1.4, 1.8, 2.0
Steering scalar $\alpha$ (Other methods, Llama)	0.1, 0.5, 1, 2.5, 5, 10
Steering scalar $\alpha$ (Other methods, Gemma)	0.1, 1, 10, 100, 1000, 10000, 100000
PCA top- $k$ components	1, 4, 8, 16, 32
Zhong top- $k$ dimensions	200, 400, 600

Appendix

Table 4: Hyperparameter search space. Tuning is conducted on Chinese, Spanish, French, and German for every dataset and model combination.

Dataset	Method	Selected hyperparameters
FLORES+	PCA	layer 25, $\alpha$ =1000, top- $k$ =1
	LDA	layer 22, $\alpha$ =1000
	Zhong (Mono)	layer 22, $\alpha$ =0.6, top- $k$ =200, anchor=15
	Zhong (Para)	layer 20, $\alpha$ =0.6, top- $k$ =200
	SAE-DM	layer 17, $\alpha$ =1000
	Gated-DM	layer 10, $\alpha$ =1000

Appendix

Table 5: Selected hyperparameters for Gemma-3-1B.

Dataset	Method	Selected hyperparameters
FLORES+	PCA	layer 29, $\alpha$ =100000, top- $k$ =16
	LDA	layer 19, $\alpha$ =1000
	Zhong (Mono)	layer 29, $\alpha$ =0.6, top- $k$ =200, anchor=12
	Zhong (Para)	layer 29, $\alpha$ =0.6, top- $k$ =200
	SAE-DM	layer 29, $\alpha$ =10000
	Gated-DM	layer 13, $\alpha$ =10000

Appendix

Table 6: Selected hyperparameters for Gemma-3-4B.

Dataset	Method	Selected hyperparameters
FLORES+	PCA	layer 31, $\alpha$ =10, top- $k$ =1
	LDA	layer 15, $\alpha$ =2.5
	Zhong (Mono)	layer 31, $\alpha$ =1.0, top- $k$ =200, anchor=15
	Zhong (Para)	layer 31, $\alpha$ =1.0, top- $k$ =200
	SAE-DM	layer 24, $\alpha$ =5
	Gated-DM	layer 21, $\alpha$ =10

Appendix

Table 7: Selected hyperparameters for Llama-3.1-8B.

Panel of Figure 6 — Figure 6: Two language-agnostic features from Llama 3.1 8B at layer 18 using the llamascope-res-32k SAE. Panels show logit effects, activation density, and top activating examples from Neuronpedia ( Lin, 2023 ) . Neither increases target-language token logits, and both track generic textual structure rather than language identity. Feature pages: 2618 , 3157 .

Figure 7: Language-specific feature count versus frequency threshold

\tau

. Feature counts stabilize for

\tau\geq 0.8

Figure 8: Non-language-specific feature activations across evaluation languages. These features activate broadly and uniformly across languages, with English slightly elevated.

Panel of Figure 9 — Figure 9: Layer-wise analyses on Llama 3.1 8B. (a) Feature density dips in middle layers and recovers in later layers. (b) Activation magnitudes peak in later layers, indicating stronger language specialization near the output.

Panel of Figure 10 — Figure 10: Convergence of intermediate feature sets (before feature overlap removal) with increasing sample size on Llama 3.1 8B. (a) Language-agnostic features (identified from random-token sequences) and (b) language-consistent features (identified from target-language sentences) both stabilize by ${\sim}$ 100 sentences, confirming that LangFIR’s inputs to the filtering step are themselves stable with limited data.

Panel of Figure 11 — Figure 11: Change in cross-entropy after ablating the top-2 language-specific features per layer on Llama 3.1 8B. Consistent with Figure 4 , ablation selectively increases loss for the target language.

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.9	1.1	4.9	0.8	0.1	0.3	0.4	0.2	0.4	0.5	0.2	0.3	0.8
	Prompt Only	20.3	23.7	23.8	27.0	29.1	36.8	25.6	14.7	15.9	7.8	15.5	26.4	22.2
	PCA	1.0	1.5	8.3	1.3	0.4	0.2	0.6	1.7	2.1	0.1	0.6	0.9	1.6
	LDA	3.2	5.0	8.3	1.8	1.1	1.7	0.3	0.2	0.9	0.7	0.1	1.1	2.0
	Zhong (Mono)	2.3	4.2	6.5	3.4	1.8	6.5	4.5	0.7	2.1	0.9	2.4	4.2	3.3
	Zhong (Para)	14.2	14.9	12.7	9.1	10.2	21.1	12.0	0.7	2.9	1.2	4.1	10.6	9.5

Appendix

Table 8: ACC

\times

BLEU averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	Prompt Only	17.9	25.3	16.8	14.1	29.1	26.8	26.3	7.5	10.4	7.3	18.0	15.1	17.9
	PCA	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	LDA	0.0	0.4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	Zhong (Mono)	2.4	3.6	1.8	1.7	0.9	5.5	5.1	0.7	0.7	0.3	3.0	2.3	2.3
	Zhong (Para)	7.4	10.2	9.1	6.4	7.3	15.9	9.4	0.3	1.4	0.6	4.2	4.7	6.4

Appendix

Table 9: ACC

\times

BLEU scores on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	2.6	2.4	9.4	2.3	0.4	0.8	0.6	0.6	1.2	1.5	0.6	0.9	1.9
	Prompt Only	34.4	34.1	29.6	28.8	32.3	39.0	25.4	17.4	22.2	8.2	15.3	33.1	26.6
	PCA	1.8	2.3	9.8	1.3	0.4	0.2	0.0	1.7	2.1	0.1	0.6	0.9	1.8
	LDA	1.3	2.1	3.7	2.1	1.4	0.1	0.0	0.2	0.9	0.7	0.1	1.7	1.2
	Zhong (Mono)	2.8	5.7	9.2	3.4	3.5	8.6	3.0	0.8	5.5	2.1	3.0	5.1	4.4
	Zhong (Para)	16.0	13.6	13.6	9.4	9.1	22.6	14.0	0.8	5.8	1.8	5.8	9.5	10.2

Appendix

Table 10: ACC

\times

BLEU scores on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	1.0	5.4	0.0	0.0	0.0	0.7	0.0	0.0	0.0	0.0	0.0	0.6
	Prompt Only	8.5	11.7	24.9	38.0	25.7	44.6	25.1	19.2	15.2	7.9	13.3	30.9	22.1
	PCA	0.1	0.8	6.8	0.0	0.0	0.0	0.6	0.0	0.0	0.0	0.0	0.0	0.7
	LDA	5.0	12.6	13.0	1.4	0.7	3.4	0.6	0.0	0.0	0.0	0.0	0.6	3.1
	Zhong (Mono)	1.7	3.4	8.6	5.1	1.1	5.3	5.4	0.5	0.1	0.2	1.1	5.1	3.1
	Zhong (Para)	19.2	20.9	15.4	11.4	14.1	24.7	12.5	1.1	1.6	1.1	2.3	17.7	11.8

Appendix

Table 11: ACC

\times

BLEU scores on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	1.7	2.7	13.7	1.3	0.7	0.3	1.0	0.7	1.0	1.0	0.3	0.3	2.1
	Prompt Only	69.3	71.0	92.3	81.7	84.3	88.0	94.3	87.3	77.3	79.3	74.7	89.7	82.4
	PCA	3.0	3.0	24.0	3.0	2.0	1.0	2.0	4.0	4.0	2.0	1.0	1.0	4.2
	LDA	14.0	12.7	43.5	4.5	3.5	5.5	3.0	2.0	3.0	3.0	1.0	6.0	8.5
	Zhong (Mono)	35.0	33.7	51.7	19.3	14.0	37.7	35.7	23.3	12.7	14.3	30.3	39.0	28.9
	Zhong (Para)	67.3	72.7	73.0	54.0	51.3	69.7	69.0	19.3	39.3	24.3	37.0	61.0	53.2

Appendix

Table 12: ACC scores (%) averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	Prompt Only	92.0	93.0	87.0	76.0	90.0	86.0	95.0	83.0	80.0	76.0	78.0	90.0	85.5
	PCA	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	LDA	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.1
	Zhong (Mono)	59.0	34.0	20.0	30.0	15.0	47.0	36.0	42.0	24.0	24.0	46.0	34.0	34.2
	Zhong (Para)	77.0	62.0	66.0	57.0	44.0	72.0	53.0	23.0	41.0	32.0	39.0	54.0	51.7

Appendix

Table 13: ACC scores (%) on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	5.0	5.0	21.0	4.0	2.0	1.0	1.0	2.0	3.0	3.0	1.0	1.0	4.1
	Prompt Only	85.0	86.0	91.0	77.0	84.0	84.0	90.0	81.0	71.0	72.0	65.0	85.0	80.9
	PCA	5.0	4.0	23.0	3.0	2.0	1.0	0.0	4.0	4.0	2.0	1.0	1.0	4.2
	LDA	15.0	5.0	32.0	6.0	4.0	4.0	1.0	2.0	3.0	3.0	1.0	9.0	7.1
	Zhong (Mono)	17.0	28.0	61.0	12.0	13.0	26.0	14.0	13.0	12.0	15.0	20.0	24.0	21.3
	Zhong (Para)	58.0	68.0	67.0	47.0	49.0	62.0	66.0	17.0	38.0	19.0	33.0	55.0	48.2

Appendix

Table 14: ACC scores (%) on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	3.0	20.0	0.0	0.0	0.0	2.0	0.0	0.0	0.0	0.0	0.0	2.1
	Prompt Only	31.0	34.0	99.0	92.0	79.0	94.0	98.0	98.0	81.0	90.0	81.0	94.0	80.9
	PCA	1.0	2.0	25.0	0.0	0.0	0.0	2.0	0.0	0.0	0.0	0.0	0.0	2.5
	LDA	13.0	32.0	55.0	3.0	3.0	7.0	5.0	0.0	0.0	0.0	0.0	3.0	10.1
	Zhong (Mono)	29.0	39.0	74.0	16.0	14.0	40.0	57.0	15.0	2.0	4.0	25.0	59.0	31.2
	Zhong (Para)	67.0	88.0	86.0	58.0	61.0	75.0	88.0	18.0	39.0	22.0	39.0	74.0	59.6

Appendix

Table 15: ACC scores (%) on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	17.5	27.1	23.8	19.1	7.0	27.0	31.2	9.3	13.8	16.2	19.5	29.6	20.1
	Prompt Only	29.1	33.7	25.7	32.4	34.5	41.7	27.2	16.7	21.0	9.9	21.0	29.5	26.9
	PCA	21.7	48.3	35.0	43.1	21.1	21.4	28.7	41.7	51.7	6.6	58.6	88.7	38.9
	LDA	23.9	39.4	17.6	40.9	29.5	25.0	7.1	11.2	30.7	23.4	5.3	18.6	22.7
	Zhong (Mono)	8.9	13.2	11.9	21.9	13.5	19.4	15.0	3.8	18.5	6.4	8.6	12.3	12.8
	Zhong (Para)	22.0	20.1	17.3	17.0	19.4	30.5	17.7	4.1	7.6	5.4	11.4	16.6	15.8

Appendix

Table 16: BLEU scores averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	Prompt Only	19.5	27.2	19.3	18.6	32.4	31.2	27.7	9.0	13.0	9.6	23.1	16.8	20.6
	PCA	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	LDA	0.0	38.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3.2
	Zhong (Mono)	4.1	10.7	9.1	5.5	6.0	11.8	14.2	1.6	2.8	1.3	6.4	6.8	6.7
	Zhong (Para)	9.6	16.5	13.8	11.3	16.7	22.1	17.7	1.3	3.4	1.9	10.8	8.7	11.1

Appendix

Table 17: BLEU scores on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	52.6	47.6	44.7	57.3	21.1	80.9	57.6	27.9	41.3	48.6	58.6	88.7	52.2
	Prompt Only	40.5	39.6	32.6	37.4	38.5	46.4	28.2	21.5	31.3	11.3	23.5	38.9	32.5
	PCA	36.7	56.9	42.7	43.1	21.1	21.4	0.0	41.7	51.7	6.6	58.6	88.7	39.1
	LDA	9.0	41.0	11.5	35.3	34.7	2.1	1.5	11.2	30.7	23.4	5.3	18.4	18.7
	Zhong (Mono)	16.5	20.3	15.2	28.3	27.0	33.3	21.3	6.2	45.8	14.0	14.8	21.3	22.0
	Zhong (Para)	27.7	20.1	20.3	20.1	18.5	36.5	21.2	4.9	15.2	9.4	17.5	17.2	19.0

Appendix

Table 18: BLEU scores on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Model	Method	es	fr	de	it	id	pt	vi	pl	cs	tr	zh	nl	Avg
Gemma 3 1B	No Steering	0.0	33.7	26.9	0.0	0.0	0.0	35.9	0.0	0.0	0.0	0.0	0.0	8.0
	Prompt Only	27.4	34.4	25.1	41.3	32.6	47.4	25.6	19.5	18.8	8.8	16.4	32.9	27.5
	PCA	6.6	39.7	27.2	0.0	0.0	0.0	28.7	0.0	0.0	0.0	0.0	0.0	8.5
	LDA	38.8	39.2	23.6	46.5	24.3	47.9	12.8	0.0	0.0	0.0	0.0	18.7	21.0
	Zhong (Mono)	6.0	8.7	11.6	31.9	7.5	13.2	9.5	3.4	6.8	4.0	4.6	8.7	9.7
	Zhong (Para)	28.7	23.7	17.9	19.7	23.0	32.9	14.2	6.1	4.2	4.8	5.9	23.9	17.1

Appendix

Table 19: BLEU scores on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .

Panel of Figure 12 — Figure 12: Ablation of intervention layer and scaling coefficient $\alpha$ on Llama 3.1 8B. (a) ACC $\times$ BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate $\alpha$ balances steering strength and translation quality.

Panel of Figure 13 — Figure 13: Ablation of frequency threshold $\tau$ and length ratio between random-token sequences and target-language sentences on Llama 3.1 8B. (a) Performance trends upward with increasing $\tau$ and stabilizes around $\tau=0.8$ . (b) Steering metrics remain stable across length ratios, showing that exact length matching is not required.

Metric	Mean	Std
ACC (%)	74.58	0.000
BLEU	23.06	0.036
ACC $\times$ BLEU	18.99	0.025

Appendix

Table 20: Steering metrics stability across 3 random seeds on Llama 3.1 8B. Near-zero standard deviations indicate robustness to random seeds.

Panel of Figure 14 — Figure 14: Ablation of number of top- $k$ language-specific features, sample size, and feature-overlap removal on Gemma 3 4B. (a) Even $k{=}2$ achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics significantly.

Panel of Figure 15 — Figure 15: Ablation of intervention layer and scaling coefficient $\alpha$ on Gemma 3 4B. (a) ACC $\times$ BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate $\alpha$ balances steering strength and translation quality.

Panel of Figure 16 — Figure 16: Ablation of frequency threshold $\tau$ and length ratio between random-token sequences and target-language sentences on Gemma 3 4B. (a) Performance generally trends upward with increasing $\tau$ and stabilizes around $\tau=0.8$ . (b) Steering metrics remain generally stable across length ratios, showing that exact length matching is not required.

Metric	Mean	Std
ACC (%)	69.86	1.339
BLEU	25.67	0.317
ACC $\times$ BLEU	20.47	0.040

Appendix

Table 21: Steering metrics stability across 3 random seeds on Gemma 3 4B. Small standard deviations indicate robustness to random seeds.

Panel of Figure 17 — Figure 17: Ablation of number of top- $k$ language-specific features, sample size, and feature-overlap removal on Gemma 3 1B. (a) Even $k{=}2$ achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics dramatically.

Panel of Figure 18 — Figure 18: Ablation of intervention layer and scaling coefficient $\alpha$ on Gemma 3 1B. (a) ACC $\times$ BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate $\alpha$ balances steering strength and translation quality.

Panel of Figure 19 — Figure 19: Ablation of frequency threshold $\tau$ and length ratio between random-token sequences and target-language sentences on Gemma 3 1B. (a) Performance trends upward with increasing $\tau$ and stabilizes around $\tau=0.8$ . (b) Steering metrics remain stable across length ratios, showing that exact length matching is not required.

Metric	Mean	Std
ACC (%)	63.06	1.273
BLEU	15.77	0.292
ACC $\times$ BLEU	10.30	0.066

Appendix

Table 22: Steering metrics stability across 3 random seeds on Gemma 3 1B. Small standard deviations indicate robustness to random seeds.

Model	Filter Negative Set	ACC (%)	BLEU	ACC $\times$ BLEU
Llama 3.1 8B	Random Tokens	78.8	23.2	18.9
	Other Languages	80.0	25.6	19.6
	Non-parallel English	81.2	23.5	19.6
Gemma 3 4B	Random Tokens	68.8	26.1	19.7
	Other Languages	72.1	25.3	20.5
	Non-parallel English	54.6	24.3	16.4

Appendix

Table 23: Ablation of filter negative set on Llama 3.1 8B, Gemma 3 4B, and Gemma 3 1B. The best result per metric within each model is bolded . Random tokens achieve competitive performance without requiring any additional language data.

Model	Method	Selected hyperparameters
Gemma-3-1B	PCA	layer 12, $\alpha$ =10000, top- $k$ =1
	LDA	layer 25, $\alpha$ =0.1
	Zhong (Mono)	layer 20, $\alpha$ =0.6, top- $k$ =200, anchor=9
	Zhong (Para)	layer 20, $\alpha$ =0.6, top- $k$ =200
	SAE-DM	layer 10, $\alpha$ =1000
	Gated-DM	layer 10, $\alpha$ =1000

Appendix

Table 24: Selected hyperparameters for the cross-lingual answer generation task across all three models.

	Non-steering		Steering
Model	No Steering	Prompt Only	PCA	LDA	Zhong (M)	Zhong (P)	SAE-DM	Gated-DM	DiffMean	LangFIR
Gemma 3 1B	7.8	7.2	5.8	7.8	6.8	8.2	7.0	7.8	7.0	6.8
Gemma 3 4B	8.8	3.8	9.8	8.8	6.2	6.0	9.8	9.0	9.2	10.0
Llama 3.1 8B	6.8	6.2	5.8	6.6	5.0	5.2	5.8	9.8	9.8	8.2
Avg	7.8	5.7	7.1	7.7	6.0	6.5	7.5	8.9	8.7	8.3

Appendix

Table 25: Cross-lingual answer generation results: EM averaged over five target languages (es, de, vi, tr, zh). The best result among steering methods within each model is bolded .

Model	Method	es	de	vi	tr	zh	Avg
Gemma 3 1B	No Steering	10.1	14.9	15.5	11.9	10.9	12.7
	Prompt Only	9.9	14.5	15.0	11.1	10.8	12.3
	PCA	8.4	12.8	12.5	10.0	9.1	10.6
	LDA	10.1	14.9	15.5	11.9	10.9	12.7
	Zhong (Mono)	10.5	13.4	9.6	11.0	11.3	11.1
	Zhong (Para)	14.1	19.4	12.3	13.4	16.3	15.1

Appendix

Table 26: Per-language F1 scores for Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B. Best result among steering methods within each model is bolded .

Model	Method	es	de	vi	tr	zh	Avg
Gemma 3 1B	No Steering	7.0	12.0	9.0	8.0	3.0	7.8
	Prompt Only	6.0	11.0	8.0	8.0	3.0	7.2
	PCA	5.0	9.0	6.0	7.0	2.0	5.8
	LDA	7.0	12.0	9.0	8.0	3.0	7.8
	Zhong (Mono)	7.0	11.0	6.0	8.0	2.0	6.8
	Zhong (Para)	7.0	15.0	7.0	9.0	3.0	8.2

Appendix

Table 27: Per-language EM scores for Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B. Best result among steering methods within each model is bolded .

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 279 papers

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
May 21, 2026Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel +3Multilingual Language Models Language Model Steering
Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +1
Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference
Aug 5, 2026Hongsheng Wang, Philipp KoehnMultilingual Language Models Language Model Steering
Johns Hopkins University, Baltimore / USA
Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
Jul 15, 2025Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani +3Multilingual Language Models Representation Learning
Faculty of Computer Science, Universitas Indonesia · Department of Natural Language Processing, MBZUAI

May 21, 2026J/K move · Enter open · S save

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel, Josef van Genabith+2

Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN)+1

Multilingual Language Models Language Model Steering Sparse Autoencoders Cross-Lingual Transfer

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) enable feature-level mechanistic interpretability and activation steering in large language models (LLMs), but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an a priori steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization (CrossSumm), using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.

Abstract

Language Model Steering

Explore similar work

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages