Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

Séverin Baroudi; Yanis Labrak; Pierfrancesco Melucci; Sergio Burdisso; Petr Motlicek; Hervé Bredin; Mirco Ravanelli; Ricard Marxer

cs.CLOct 3, 2026

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

Authors: Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci, Sergio Burdisso, Petr Motlicek, Hervé Bredin, Mirco Ravanelli, Ricard Marxer

Organizations: Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland · Sapienza University of Rome, Italy ·

Abstract

Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.

Figures & tables

Panel of Figure 1 — Figure 1: Method overview: (a) extraction of a task direction (ASR in this case) from contrasting instructions on the same audio; (b) steering at inference: the direction is added at a single layer $\ell^{\star}$ to every token, audio positions included.

Method			LibriSpeech (WER $\downarrow$ )				CommonVoice (CER $\downarrow$ )					IEMOCAP	AudioCaps
Prompt	Task Steering	LoRA SFT	dev clean	dev other	test clean	test other	en	fr	ar	hi	ru	WA $\uparrow$	R-1 $\uparrow$	CIDEr $\uparrow$
Qwen2-Audio-7B-Instruct
✓	✗	✗	2.77	5.33	2.94	5.66	7.4	9.1	52.3	51.7	22.7	70.0	34.1	0.26
✗	✓	✗	6.58	10.6	6.03	10.2	13.2	10.8	68.1	73.3	32.0	63.2	40.1	0.43
✓	✓	✗	2.43	4.87	2.52	5.16	6.9	7.9	45.9	46.6	21.3	74.2	36.0	0.30
✗	✗	✓	7.54	10.0	8.14	10.1	7.1	6.1	30.2	40.7	14.4	75.3	48.2	0.64

Table 1: Task steering results on ASR, ER and AC. Rows with a prompt are averaged over three prompts for each task. Best score for each system is reported in bold. Since our focus is solely on training-free methods, we report LoRA-SFT in grey.

Panel of Figure 2 — Figure 2: grid searches for task induction through activation addition (w/o prompt), on Qwen2-Audio-7B-Instruct , using the training set assembled in Section 3 .

Task vector	LibriSpeech	CommonVoice
injected at	WER $\downarrow$	CER $\downarrow$
Text tokens	14.90	48.8
Text + Audio tokens	8.35	39.5

Table 2: Impact of task steering over the input prompt tokens, on ASR using Qwen2-Audio-7B-Instruct . LibriSpeech WER is the mean of the four splits dev-c/o and test-c/o. CommonVoice CER is the mean over the en/fr/ar/hi/ru test sets.

	Qwen2-Audio			Phi-4-MM
Method	ar	hi	ru	ar	hi	ru
Prompt & Task Steering	45.9	46.6	21.3	100.6	37.9	91.4
+ Script Steering	40.5	44.1	21.2	102.4	36.7	62.2

Table 3: Script-Normalization direction combined with Prompt and Task Steering for CommonVoice on non-Latin scripts (Arabic, Hindi and Russian). CER is reported.

Task vector	w/o Prompt		Prompt
extracted from	LS WER $\downarrow$	CV CER $\downarrow$	LS WER $\downarrow$	CV CER $\downarrow$
LS + CV	8.35	39.5	3.75 $\pm 0.19$	25.7 $\pm 1.4$
LS	8.31	38.5	3.75 $\pm 0.18$	25.6 $\pm 1.3$
CV	8.58	38.7	3.76 $\pm 0.17$	25.5 $\pm 1.2$
No Steering	239.9	159.7	4.18 $\pm 0.49$	28.6 $\pm 2.1$
Random	196.7	178.1	4.31 $\pm 0.54$	29.7 $\pm 3.0$

Table 4: Out-of-domain transfer on Qwen2-Audio for ASR (reported on the same sets as in Table 2 ). The

\pm

values in the Prompt columns give the standard deviation over the three prompts. Random replaces the task vector with a random one of equal norm at the same site.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 278 papers

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
May 30, 2026Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng +3Language Model Steering Automatic Speech Recognition
University of Illinois Urbana-Champaign · Korea Advanced Institute of Science and Technology
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.
Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models
Mar 15, 2026Lok-Lam Ieong, Chia-Chien Chen, Chih-Kai Yang +3Language Model Steering Audio-Language Models
National Taiwan University, Taiwan

May 30, 2026J/K move · Enter open · S save

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng, Julia Hockenmaier+2

University of Illinois Urbana-Champaign · Korea Advanced Institute of Science and Technology

Language Model Steering Automatic Speech Recognition Speech Language Models

Open paper AskPDF arXiv

Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself.

Abstract

Language Model Steering

Explore similar work

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

Steer Like the LLM: Activation Steering that Mimics Prompting

Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models