Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult. Representation-level steering addresses this by adding language-specific vectors to model activations at inference time, but identifying language-specific directions in the residual stream often relies on multilingual or parallel data that can be expensive to obtain. Sparse autoencoders (SAEs) decompose residual activations into interpretable, sparse feature directions and offer a natural basis for this search, yet existing SAE-based approaches face the same data constraint. We introduce LangFIR (Language Feature Identification via Random-token Filtering), a method that discovers language-specific SAE features using only a small amount of monolingual data and random-token sequences. Many SAE features consistently activated by target-language inputs do not encode language identity. Random-token sequences surface these language-agnostic features, allowing LangFIR to filter them out and isolate a sparse set of language-specific features. We show that these features are extremely sparse, highly selective for their target language, and causally important: directional ablation increases cross-entropy loss only for the corresponding language. Using these features to construct steering vectors for the multilingual generation control task, LangFIR achieves the best average accuracy x BLEU among steering methods across three models (Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B), three datasets, and twelve target languages, outperforming the strongest monolingual baseline by up to 4.7x and surpassing methods that use parallel data. Our results suggest that language identity in multilingual LLMs is localized in a sparse set of feature directions discoverable with monolingual data. Code is available at https://github.com/JaMussCraft/LangFIR
Figures & tables
Figure 1: Overview of LangFIR. (1) For each target-language sentence, a random-token sequence is generated by sampling from the tokenizer’s vocabulary. (2) Residual-stream activations are extracted at a layer l and encoded through SAE. (3) Sample-wise filtering is performed with activation frequency threshold τ to identify features that activate frequently for target-language sentences ( language-consistent ) and for random-token sequences ( language-agnostic ). (4) Removing language-agnostic features from language-consistent features yields a sparse set of language-specific features .
Figure 2: Analyses of language-specific feature identification across sample sizes on Llama 3.1 8B. (a) Overlap between language-consistent and language-agnostic feature sets approaches 100%. (b) Fewer than 5 language-specific features remain per language. Both quantities stabilize by ∼ 100 sample size.
Figure 3: Analyses of language-specific feature properties on Llama 3.1 8B. (a) Rank-1 features dominate in mean activation magnitude. (b) Language-specific features activate selectively on their target language and remain near zero on others.
Figure 4: Change in cross-entropy after ablating the top-2 language-specific features per layer on Llama 3.1 8B. Ablation increases loss selectively for the target language. English is a notable exception.
Non-steering
Steering
Model
No Steering
Prompt Only
PCA
LDA
Zhong (M)
Zhong (P)
SAE-DM
Gated-DM
DiffMean
LangFIR
Gemma 3 1B
0.8
22.2
1.6
2.0
3.3
9.5
14.7
9.0
15.3
15.5
Gemma 3 4B
2.6
35.5
3.3
15.2
10.5
13.9
23.2
10.8
23.9
26.6
Llama 3.1 8B
1.5
34.7
2.9
22.7
6.6
7.6
4.7
22.6
22.2
23.4
Avg
1.6
30.8
2.6
13.3
6.8
10.3
14.2
14.1
20.5
21.8
Table 1: ACC × BLEU averaged over twelve target languages and three datasets (FLORES+, WikiMatrix, Tatoeba). The best result among steering methods within each model is bolded .
Figure 5: Ablation of number of top- k language-specific features, sample size, and feature-overlap removal on Llama 3.1 8B. (a) Even k=1 achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics dramatically.
Non-steering
Steering
Model
No Steering
Prompt Only
PCA
LDA
Zhong (M)
Zhong (P)
SAE-DM
Gated-DM
DiffMean
LangFIR
Gemma 3 1B
12.7
12.3
10.6
12.7
11.1
15.1
13.3
13.6
14.4
16.7
Gemma 3 4B
14.6
7.9
15.6
14.7
11.3
13.3
22.5
19.3
21.7
20.0
Llama 3.1 8B
12.3
12.2
11.6
11.8
10.6
11.2
11.1
20.8
20.7
21.6
Avg
13.2
10.8
12.6
13.0
11.0
13.2
15.6
17.9
18.9
19.4
Table 2: Cross-lingual answer generation results: F1 averaged over five target languages (es, de, vi, tr, zh). The best result among steering methods within each model is bolded .
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Model ID
SAE Release
SAE ID
google/gemma-3-1b-pt
gemma-scope-2-1b-pt-res-all
layer_{layer}_width_16k_l0_small
google/gemma-3-4b-pt
gemma-scope-2-4b-pt-res-all
layer_{layer}_width_16k_l0_small
meta-llama/Llama-3.1-8B
llama_scope_lxr_8x
l{layer}r_8x
Appendix
Table 3: Models and SAEs used in our experiments.
Hyperparameter
Values
Intervention layer percentile
0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Steering scalar α (Zhong Mono/Para)
0.2, 0.6, 1.0, 1.4, 1.8, 2.0
Steering scalar α (Other methods, Llama)
0.1, 0.5, 1, 2.5, 5, 10
Steering scalar α (Other methods, Gemma)
0.1, 1, 10, 100, 1000, 10000, 100000
PCA top- k components
1, 4, 8, 16, 32
Zhong top- k dimensions
200, 400, 600
Appendix
Table 4: Hyperparameter search space. Tuning is conducted on Chinese, Spanish, French, and German for every dataset and model combination.
Dataset
Method
Selected hyperparameters
FLORES+
PCA
layer 25, α =1000, top- k =1
LDA
layer 22, α =1000
Zhong (Mono)
layer 22, α =0.6, top- k =200, anchor=15
Zhong (Para)
layer 20, α =0.6, top- k =200
SAE-DM
layer 17, α =1000
Gated-DM
layer 10, α =1000
Appendix
Table 5: Selected hyperparameters for Gemma-3-1B.
Dataset
Method
Selected hyperparameters
FLORES+
PCA
layer 29, α =100000, top- k =16
LDA
layer 19, α =1000
Zhong (Mono)
layer 29, α =0.6, top- k =200, anchor=12
Zhong (Para)
layer 29, α =0.6, top- k =200
SAE-DM
layer 29, α =10000
Gated-DM
layer 13, α =10000
Appendix
Table 6: Selected hyperparameters for Gemma-3-4B.
Dataset
Method
Selected hyperparameters
FLORES+
PCA
layer 31, α =10, top- k =1
LDA
layer 15, α =2.5
Zhong (Mono)
layer 31, α =1.0, top- k =200, anchor=15
Zhong (Para)
layer 31, α =1.0, top- k =200
SAE-DM
layer 24, α =5
Gated-DM
layer 21, α =10
Appendix
Table 7: Selected hyperparameters for Llama-3.1-8B.
Figure 6: Two language-agnostic features from Llama 3.1 8B at layer 18 using the llamascope-res-32k SAE. Panels show logit effects, activation density, and top activating examples from Neuronpedia ( Lin, 2023 ) . Neither increases target-language token logits, and both track generic textual structure rather than language identity. Feature pages: 2618 , 3157 .
Figure 7: Language-specific feature count versus frequency threshold τ . Feature counts stabilize for τ≥0.8 .
Figure 8: Non-language-specific feature activations across evaluation languages. These features activate broadly and uniformly across languages, with English slightly elevated.
Figure 9: Layer-wise analyses on Llama 3.1 8B. (a) Feature density dips in middle layers and recovers in later layers. (b) Activation magnitudes peak in later layers, indicating stronger language specialization near the output.
Figure 10: Convergence of intermediate feature sets (before feature overlap removal) with increasing sample size on Llama 3.1 8B. (a) Language-agnostic features (identified from random-token sequences) and (b) language-consistent features (identified from target-language sentences) both stabilize by ∼ 100 sentences, confirming that LangFIR’s inputs to the filtering step are themselves stable with limited data.
Figure 11: Change in cross-entropy after ablating the top-2 language-specific features per layer on Llama 3.1 8B. Consistent with Figure 4 , ablation selectively increases loss for the target language.
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.9
1.1
4.9
0.8
0.1
0.3
0.4
0.2
0.4
0.5
0.2
0.3
0.8
Prompt Only
20.3
23.7
23.8
27.0
29.1
36.8
25.6
14.7
15.9
7.8
15.5
26.4
22.2
PCA
1.0
1.5
8.3
1.3
0.4
0.2
0.6
1.7
2.1
0.1
0.6
0.9
1.6
LDA
3.2
5.0
8.3
1.8
1.1
1.7
0.3
0.2
0.9
0.7
0.1
1.1
2.0
Zhong (Mono)
2.3
4.2
6.5
3.4
1.8
6.5
4.5
0.7
2.1
0.9
2.4
4.2
3.3
Zhong (Para)
14.2
14.9
12.7
9.1
10.2
21.1
12.0
0.7
2.9
1.2
4.1
10.6
9.5
Appendix
Table 8: ACC × BLEU averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Prompt Only
17.9
25.3
16.8
14.1
29.1
26.8
26.3
7.5
10.4
7.3
18.0
15.1
17.9
PCA
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
LDA
0.0
0.4
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Zhong (Mono)
2.4
3.6
1.8
1.7
0.9
5.5
5.1
0.7
0.7
0.3
3.0
2.3
2.3
Zhong (Para)
7.4
10.2
9.1
6.4
7.3
15.9
9.4
0.3
1.4
0.6
4.2
4.7
6.4
Appendix
Table 9: ACC × BLEU scores on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
2.6
2.4
9.4
2.3
0.4
0.8
0.6
0.6
1.2
1.5
0.6
0.9
1.9
Prompt Only
34.4
34.1
29.6
28.8
32.3
39.0
25.4
17.4
22.2
8.2
15.3
33.1
26.6
PCA
1.8
2.3
9.8
1.3
0.4
0.2
0.0
1.7
2.1
0.1
0.6
0.9
1.8
LDA
1.3
2.1
3.7
2.1
1.4
0.1
0.0
0.2
0.9
0.7
0.1
1.7
1.2
Zhong (Mono)
2.8
5.7
9.2
3.4
3.5
8.6
3.0
0.8
5.5
2.1
3.0
5.1
4.4
Zhong (Para)
16.0
13.6
13.6
9.4
9.1
22.6
14.0
0.8
5.8
1.8
5.8
9.5
10.2
Appendix
Table 10: ACC × BLEU scores on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
1.0
5.4
0.0
0.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.6
Prompt Only
8.5
11.7
24.9
38.0
25.7
44.6
25.1
19.2
15.2
7.9
13.3
30.9
22.1
PCA
0.1
0.8
6.8
0.0
0.0
0.0
0.6
0.0
0.0
0.0
0.0
0.0
0.7
LDA
5.0
12.6
13.0
1.4
0.7
3.4
0.6
0.0
0.0
0.0
0.0
0.6
3.1
Zhong (Mono)
1.7
3.4
8.6
5.1
1.1
5.3
5.4
0.5
0.1
0.2
1.1
5.1
3.1
Zhong (Para)
19.2
20.9
15.4
11.4
14.1
24.7
12.5
1.1
1.6
1.1
2.3
17.7
11.8
Appendix
Table 11: ACC × BLEU scores on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
1.7
2.7
13.7
1.3
0.7
0.3
1.0
0.7
1.0
1.0
0.3
0.3
2.1
Prompt Only
69.3
71.0
92.3
81.7
84.3
88.0
94.3
87.3
77.3
79.3
74.7
89.7
82.4
PCA
3.0
3.0
24.0
3.0
2.0
1.0
2.0
4.0
4.0
2.0
1.0
1.0
4.2
LDA
14.0
12.7
43.5
4.5
3.5
5.5
3.0
2.0
3.0
3.0
1.0
6.0
8.5
Zhong (Mono)
35.0
33.7
51.7
19.3
14.0
37.7
35.7
23.3
12.7
14.3
30.3
39.0
28.9
Zhong (Para)
67.3
72.7
73.0
54.0
51.3
69.7
69.0
19.3
39.3
24.3
37.0
61.0
53.2
Appendix
Table 12: ACC scores (%) averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Prompt Only
92.0
93.0
87.0
76.0
90.0
86.0
95.0
83.0
80.0
76.0
78.0
90.0
85.5
PCA
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
LDA
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
Zhong (Mono)
59.0
34.0
20.0
30.0
15.0
47.0
36.0
42.0
24.0
24.0
46.0
34.0
34.2
Zhong (Para)
77.0
62.0
66.0
57.0
44.0
72.0
53.0
23.0
41.0
32.0
39.0
54.0
51.7
Appendix
Table 13: ACC scores (%) on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
5.0
5.0
21.0
4.0
2.0
1.0
1.0
2.0
3.0
3.0
1.0
1.0
4.1
Prompt Only
85.0
86.0
91.0
77.0
84.0
84.0
90.0
81.0
71.0
72.0
65.0
85.0
80.9
PCA
5.0
4.0
23.0
3.0
2.0
1.0
0.0
4.0
4.0
2.0
1.0
1.0
4.2
LDA
15.0
5.0
32.0
6.0
4.0
4.0
1.0
2.0
3.0
3.0
1.0
9.0
7.1
Zhong (Mono)
17.0
28.0
61.0
12.0
13.0
26.0
14.0
13.0
12.0
15.0
20.0
24.0
21.3
Zhong (Para)
58.0
68.0
67.0
47.0
49.0
62.0
66.0
17.0
38.0
19.0
33.0
55.0
48.2
Appendix
Table 14: ACC scores (%) on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
3.0
20.0
0.0
0.0
0.0
2.0
0.0
0.0
0.0
0.0
0.0
2.1
Prompt Only
31.0
34.0
99.0
92.0
79.0
94.0
98.0
98.0
81.0
90.0
81.0
94.0
80.9
PCA
1.0
2.0
25.0
0.0
0.0
0.0
2.0
0.0
0.0
0.0
0.0
0.0
2.5
LDA
13.0
32.0
55.0
3.0
3.0
7.0
5.0
0.0
0.0
0.0
0.0
3.0
10.1
Zhong (Mono)
29.0
39.0
74.0
16.0
14.0
40.0
57.0
15.0
2.0
4.0
25.0
59.0
31.2
Zhong (Para)
67.0
88.0
86.0
58.0
61.0
75.0
88.0
18.0
39.0
22.0
39.0
74.0
59.6
Appendix
Table 15: ACC scores (%) on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
17.5
27.1
23.8
19.1
7.0
27.0
31.2
9.3
13.8
16.2
19.5
29.6
20.1
Prompt Only
29.1
33.7
25.7
32.4
34.5
41.7
27.2
16.7
21.0
9.9
21.0
29.5
26.9
PCA
21.7
48.3
35.0
43.1
21.1
21.4
28.7
41.7
51.7
6.6
58.6
88.7
38.9
LDA
23.9
39.4
17.6
40.9
29.5
25.0
7.1
11.2
30.7
23.4
5.3
18.6
22.7
Zhong (Mono)
8.9
13.2
11.9
21.9
13.5
19.4
15.0
3.8
18.5
6.4
8.6
12.3
12.8
Zhong (Para)
22.0
20.1
17.3
17.0
19.4
30.5
17.7
4.1
7.6
5.4
11.4
16.6
15.8
Appendix
Table 16: BLEU scores averaged over FLORES+, WikiMatrix, and Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Prompt Only
19.5
27.2
19.3
18.6
32.4
31.2
27.7
9.0
13.0
9.6
23.1
16.8
20.6
PCA
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
LDA
0.0
38.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
3.2
Zhong (Mono)
4.1
10.7
9.1
5.5
6.0
11.8
14.2
1.6
2.8
1.3
6.4
6.8
6.7
Zhong (Para)
9.6
16.5
13.8
11.3
16.7
22.1
17.7
1.3
3.4
1.9
10.8
8.7
11.1
Appendix
Table 17: BLEU scores on FLORES+ for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
52.6
47.6
44.7
57.3
21.1
80.9
57.6
27.9
41.3
48.6
58.6
88.7
52.2
Prompt Only
40.5
39.6
32.6
37.4
38.5
46.4
28.2
21.5
31.3
11.3
23.5
38.9
32.5
PCA
36.7
56.9
42.7
43.1
21.1
21.4
0.0
41.7
51.7
6.6
58.6
88.7
39.1
LDA
9.0
41.0
11.5
35.3
34.7
2.1
1.5
11.2
30.7
23.4
5.3
18.4
18.7
Zhong (Mono)
16.5
20.3
15.2
28.3
27.0
33.3
21.3
6.2
45.8
14.0
14.8
21.3
22.0
Zhong (Para)
27.7
20.1
20.3
20.1
18.5
36.5
21.2
4.9
15.2
9.4
17.5
17.2
19.0
Appendix
Table 18: BLEU scores on WikiMatrix for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Model
Method
es
fr
de
it
id
pt
vi
pl
cs
tr
zh
nl
Avg
Gemma 3 1B
No Steering
0.0
33.7
26.9
0.0
0.0
0.0
35.9
0.0
0.0
0.0
0.0
0.0
8.0
Prompt Only
27.4
34.4
25.1
41.3
32.6
47.4
25.6
19.5
18.8
8.8
16.4
32.9
27.5
PCA
6.6
39.7
27.2
0.0
0.0
0.0
28.7
0.0
0.0
0.0
0.0
0.0
8.5
LDA
38.8
39.2
23.6
46.5
24.3
47.9
12.8
0.0
0.0
0.0
0.0
18.7
21.0
Zhong (Mono)
6.0
8.7
11.6
31.9
7.5
13.2
9.5
3.4
6.8
4.0
4.6
8.7
9.7
Zhong (Para)
28.7
23.7
17.9
19.7
23.0
32.9
14.2
6.1
4.2
4.8
5.9
23.9
17.1
Appendix
Table 19: BLEU scores on Tatoeba for Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B. The best result among steering methods per language within each model is bolded .
Figure 12: Ablation of intervention layer and scaling coefficient α on Llama 3.1 8B. (a) ACC × BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate α balances steering strength and translation quality.
Figure 13: Ablation of frequency threshold τ and length ratio between random-token sequences and target-language sentences on Llama 3.1 8B. (a) Performance trends upward with increasing τ and stabilizes around τ=0.8 . (b) Steering metrics remain stable across length ratios, showing that exact length matching is not required.
Metric
Mean
Std
ACC (%)
74.58
0.000
BLEU
23.06
0.036
ACC × BLEU
18.99
0.025
Appendix
Table 20: Steering metrics stability across 3 random seeds on Llama 3.1 8B. Near-zero standard deviations indicate robustness to random seeds.
Figure 14: Ablation of number of top- k language-specific features, sample size, and feature-overlap removal on Gemma 3 4B. (a) Even k=2 achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics significantly.
Figure 15: Ablation of intervention layer and scaling coefficient α on Gemma 3 4B. (a) ACC × BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate α balances steering strength and translation quality.
Figure 16: Ablation of frequency threshold τ and length ratio between random-token sequences and target-language sentences on Gemma 3 4B. (a) Performance generally trends upward with increasing τ and stabilizes around τ=0.8 . (b) Steering metrics remain generally stable across length ratios, showing that exact length matching is not required.
Metric
Mean
Std
ACC (%)
69.86
1.339
BLEU
25.67
0.317
ACC × BLEU
20.47
0.040
Appendix
Table 21: Steering metrics stability across 3 random seeds on Gemma 3 4B. Small standard deviations indicate robustness to random seeds.
Figure 17: Ablation of number of top- k language-specific features, sample size, and feature-overlap removal on Gemma 3 1B. (a) Even k=2 achieves strong performance. (b) Steering metrics are stable with as few as 10 sentences. (c) Removing feature-overlap removal degrades all steering metrics dramatically.
Figure 18: Ablation of intervention layer and scaling coefficient α on Gemma 3 1B. (a) ACC × BLEU peaks in middle layers, where steering redirects language without disrupting content. (b) Intermediate α balances steering strength and translation quality.
Figure 19: Ablation of frequency threshold τ and length ratio between random-token sequences and target-language sentences on Gemma 3 1B. (a) Performance trends upward with increasing τ and stabilizes around τ=0.8 . (b) Steering metrics remain stable across length ratios, showing that exact length matching is not required.
Metric
Mean
Std
ACC (%)
63.06
1.273
BLEU
15.77
0.292
ACC × BLEU
10.30
0.066
Appendix
Table 22: Steering metrics stability across 3 random seeds on Gemma 3 1B. Small standard deviations indicate robustness to random seeds.
Model
Filter Negative Set
ACC (%)
BLEU
ACC × BLEU
Llama 3.1 8B
Random Tokens
78.8
23.2
18.9
Other Languages
80.0
25.6
19.6
Non-parallel English
81.2
23.5
19.6
Gemma 3 4B
Random Tokens
68.8
26.1
19.7
Other Languages
72.1
25.3
20.5
Non-parallel English
54.6
24.3
16.4
Appendix
Table 23: Ablation of filter negative set on Llama 3.1 8B, Gemma 3 4B, and Gemma 3 1B. The best result per metric within each model is bolded . Random tokens achieve competitive performance without requiring any additional language data.
Model
Method
Selected hyperparameters
Gemma-3-1B
PCA
layer 12, α =10000, top- k =1
LDA
layer 25, α =0.1
Zhong (Mono)
layer 20, α =0.6, top- k =200, anchor=9
Zhong (Para)
layer 20, α =0.6, top- k =200
SAE-DM
layer 10, α =1000
Gated-DM
layer 10, α =1000
Appendix
Table 24: Selected hyperparameters for the cross-lingual answer generation task across all three models.
Non-steering
Steering
Model
No Steering
Prompt Only
PCA
LDA
Zhong (M)
Zhong (P)
SAE-DM
Gated-DM
DiffMean
LangFIR
Gemma 3 1B
7.8
7.2
5.8
7.8
6.8
8.2
7.0
7.8
7.0
6.8
Gemma 3 4B
8.8
3.8
9.8
8.8
6.2
6.0
9.8
9.0
9.2
10.0
Llama 3.1 8B
6.8
6.2
5.8
6.6
5.0
5.2
5.8
9.8
9.8
8.2
Avg
7.8
5.7
7.1
7.7
6.0
6.5
7.5
8.9
8.7
8.3
Appendix
Table 25: Cross-lingual answer generation results: EM averaged over five target languages (es, de, vi, tr, zh). The best result among steering methods within each model is bolded .
Model
Method
es
de
vi
tr
zh
Avg
Gemma 3 1B
No Steering
10.1
14.9
15.5
11.9
10.9
12.7
Prompt Only
9.9
14.5
15.0
11.1
10.8
12.3
PCA
8.4
12.8
12.5
10.0
9.1
10.6
LDA
10.1
14.9
15.5
11.9
10.9
12.7
Zhong (Mono)
10.5
13.4
9.6
11.0
11.3
11.1
Zhong (Para)
14.1
19.4
12.3
13.4
16.3
15.1
Appendix
Table 26: Per-language F1 scores for Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B. Best result among steering methods within each model is bolded .
Model
Method
es
de
vi
tr
zh
Avg
Gemma 3 1B
No Steering
7.0
12.0
9.0
8.0
3.0
7.8
Prompt Only
6.0
11.0
8.0
8.0
3.0
7.2
PCA
5.0
9.0
6.0
7.0
2.0
5.8
LDA
7.0
12.0
9.0
8.0
3.0
7.8
Zhong (Mono)
7.0
11.0
6.0
8.0
2.0
6.8
Zhong (Para)
7.0
15.0
7.0
9.0
3.0
8.2
Appendix
Table 27: Per-language EM scores for Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B. Best result among steering methods within each model is bolded .
Sparse autoencoders (SAEs) enable feature-level mechanistic interpretability and activation steering in large language models (LLMs), but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an a priori steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization (CrossSumm), using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.
Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel +3
Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +1
Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show macro-average accuracy improvements of 11.28 percentage points on XCOPA, 4.11 points on XNLI, and 7.84 points on PAWS-X. Additional evaluation with Qwen2.5-7B-Instruct yields a macro-average improvement of 3.33 percentage points on XCOPA.
Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to late layers of the model and are interpretable. These features influence the model's multilingual performance and language output, and can be used for language identification with performance comparable to fastText, along with more interpretability. Our code and complete figures are available at https://github.com/LyzanderAndrylie/language-specific-features.