Preference-Aware Pareto Steering for Multi-Task Speech LLMs
Authors: Liuyuan Jiang, Vishal Sunder, Samuel Thomas, Brian Kingsbury, Jeff Kuo, Lisha Chen, George Saon
Organizations: Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA · IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA
Speech large language models (LLMs) perform multiple tasks, including automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST). Joint training of speech LLMs creates task trade-offs, yet standard training with fixed loss weights does not generally control the resulting trade-off according to deployment preferences. To address this, we propose Endpoint-Anchored Preference-Aware Sampling Uniformly along the Pareto Front (EA-SURF), an a priori method that adapts scalarization weights so that the resulting task trade-off reflects the desired preference. EA-SURF retains the standard weighted-loss training pipeline, tracking the current model's performance relative to each task-specific reference and adapting the scalarization weights as training proceeds. We theoretically show that EA-SURF can match a desired preference, with the update becoming stationary at the target under checkable regularity conditions. Across six three-objective speech LLM requests, EA-SURF empirically converges toward the requested targets and increases endpoint-inclusive normalized hypervolume by 17.8% over fixed loss weighting.
Figures & tables
Figure 1: EA-SURF preference steering for ASR, timestamp prediction (TS), and speech translation (ST). Specialist models anchor the objective-space geometry, while a deployment preference p is mapped by ( 3 ) to target specialist-relative distance shares q⋆(p) . EA-SURF iteratively adapts the scalarization weights w from the measured share mismatch and continues training from the current model toward the requested operating point.
Figure 2: EA-SURF dynamics for p=(.4,.4,.2) . Left: attained shares in the simplex. Upper right: current-to-target ratios. Lower right: the resulting adaptive LS weights. The gold star marks q⋆(p)=(.3,.3,.4) , and the dashed line marks ratio one.
Fixed LS
EA-SURF
Preference
WER (%) ↓
AAS (ms) ↓
WER (%) ↓
AAS (ms) ↓
(.25,.75)
7.37
32.8
7.52
30.6
(.50,.50)
7.26
35.9
7.34
34.5
(.75,.25)
7.24
41.7
7.30
40.1
Table 1: Bi-objective results from random initialization. The first two rows are fixed-LS specialists reused as EA-SURF anchors. The remaining rows compare the interior preferences.
ASR WER (%) ↓
AAS (ms) ↓
BLEU ↑
Request p
LS
EA
LS
EA
LS
EA
(.2,.2,.6)
7.38
7.49
54.3
41.8
32.89
32.54
(.2,.4,.4)
7.31
7.59
43.0
35.2
31.91
31.64
(.2,.6,.2)
7.33
7.62
38.0
35.8
29.54
31.71
(.4,.2,.4)
7.20
7.24
50.9
38.8
32.76
32.53
(.4,.4,.2)
7.23
7.39
42.6
37.0
30.04
31.39
Table 2: Three-objective results with 180k-step training budgets for both methods. EA denotes EA-SURF and Δ=EA-SURF−LS .
Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.
Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself.
Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs.