Preference-Aware Pareto Steering for Multi-Task Speech LLMs

Liuyuan Jiang; Vishal Sunder; Samuel Thomas; Brian Kingsbury; Jeff Kuo; Lisha Chen; George Saon

cs.CLOct 19, 2026

Preference-Aware Pareto Steering for Multi-Task Speech LLMs

Authors: Liuyuan Jiang, Vishal Sunder, Samuel Thomas, Brian Kingsbury, Jeff Kuo, Lisha Chen, George Saon

Organizations: Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA · IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA

Abstract

Speech large language models (LLMs) perform multiple tasks, including automatic speech recognition (ASR), timestamp prediction (TS), and speech translation (ST). Joint training of speech LLMs creates task trade-offs, yet standard training with fixed loss weights does not generally control the resulting trade-off according to deployment preferences. To address this, we propose Endpoint-Anchored Preference-Aware Sampling Uniformly along the Pareto Front (EA-SURF), an a priori method that adapts scalarization weights so that the resulting task trade-off reflects the desired preference. EA-SURF retains the standard weighted-loss training pipeline, tracking the current model's performance relative to each task-specific reference and adapting the scalarization weights as training proceeds. We theoretically show that EA-SURF can match a desired preference, with the update becoming stationary at the target under checkable regularity conditions. Across six three-objective speech LLM requests, EA-SURF empirically converges toward the requested targets and increases endpoint-inclusive normalized hypervolume by 17.8% over fixed loss weighting.

Figures & tables

Figure 1: EA-SURF preference steering for ASR, timestamp prediction (TS), and speech translation (ST). Specialist models anchor the objective-space geometry, while a deployment preference $\mathbf{p}$ is mapped by ( 3 ) to target specialist-relative distance shares $\mathbf{q}^{\star}(\mathbf{p})$ . EA-SURF iteratively adapts the scalarization weights $\mathbf{w}$ from the measured share mismatch and continues training from the current model toward the requested operating point.

Figure 2: EA-SURF dynamics for

\mathbf{p}=(.4,.4,.2)

. Left: attained shares in the simplex. Upper right: current-to-target ratios. Lower right: the resulting adaptive LS weights. The gold star marks

\mathbf{q}^{\star}(\mathbf{p})=(.3,.3,.4)

, and the dashed line marks ratio one.

	Fixed LS		EA-SURF
Preference	WER (%) $\downarrow$	AAS (ms) $\downarrow$	WER (%) $\downarrow$	AAS (ms) $\downarrow$
$(.25,.75)$	7.37	32.8	7.52	30.6
$(.50,.50)$	7.26	35.9	7.34	34.5
$(.75,.25)$	7.24	41.7	7.30	40.1

Table 1: Bi-objective results from random initialization. The first two rows are fixed-LS specialists reused as EA-SURF anchors. The remaining rows compare the interior preferences.

	ASR WER (%) $\downarrow$		AAS (ms) $\downarrow$		BLEU $\uparrow$
Request $\mathbf{p}$	LS	EA	LS	EA	LS	EA
$(.2,.2,.6)$	7.38	7.49	54.3	41.8	32.89	32.54
$(.2,.4,.4)$	7.31	7.59	43.0	35.2	31.91	31.64
$(.2,.6,.2)$	7.33	7.62	38.0	35.8	29.54	31.71
$(.4,.2,.4)$	7.20	7.24	50.9	38.8	32.76	32.53
$(.4,.4,.2)$	7.23	7.39	42.6	37.0	30.04	31.39

Table 2: Three-objective results with 180k-step training budgets for both methods. EA denotes EA-SURF and

\Delta=\text{EA-SURF}-\text{LS}

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 389 papers

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition
Oct 3, 2026Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci +5Language Model Steering Speech Language Models
Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland +5
SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
May 30, 2026Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng +3Language Model Steering Automatic Speech Recognition
University of Illinois Urbana-Champaign · Korea Advanced Institute of Science and Technology
Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning
Mar 16, 2026Jingxiang Chen, Minseok Kim, Seong-Gyun Leem +13Speech Language Models Emotion Representation in Language Models
Meta Reality Labs

Oct 3, 2026J/K move · Enter open · S save

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci, Sergio Burdisso+4

Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland+5

Language Model Steering Speech Language Models Automatic Speech Recognition

Open paper AskPDF arXiv

Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.

Abstract

Multi-Task Learning

Explore similar work

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning