Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

Hongsheng Wang; Philipp Koehn

cs.CLAug 5, 2026

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

Authors: Hongsheng Wang, Philipp Koehn

Organizations: Johns Hopkins University · Baltimore / USA

Abstract

Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show average accuracy improvements of 10.9 percentage points on XCOPA, 5.3 points on XNLI, and 1.9 points on MGSM.

Explore similar work

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
May 21, 2026Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel +3Multilingual Mechanistic Interpretability
SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
May 30, 2026Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng +3Speech Large Language Models Automatic Speech Recognition Systems
DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
Jun 3, 2026Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui +3Small Language Models Chain-Of-Thought Distillation

May 21, 2026 · cs.CLJ/K move · Enter open · S save

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel, Josef van Genabith+2

Sparse autoencoders (SAEs) enable feature-level mechanistic interpretability and activation steering in large language models (LLMs), but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an \emph{a priori} steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization (CrossSumm), using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.

Multilingual Mechanistic Interpretability

Open paper Ask about it PDF

Abstract

Explore similar work

Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer