cs.CVNov 11, 2026

MAST: Medoid-Anchored Subspace Steering for Visual Grounding in Multimodal LLMs

Authors: Zhitao Chen, Zhenxin Cai, Jingyuan Li, Jin Sha

Organizations: School of Electronic Science and Engineering, Nanjing University, China · College of Electronic Science and Technology, National University of Defense Technology, China

Abstract

Existing activation steering methods for multimodal large language models (MLLMs) use the global activation mean as the steering vector, which tends to cause cross-task interference and unstable performance. We trace this failure to the multi-centered geometry of task activations. The global mean lands in the empty region between clusters and is faithful to no single task. Moreover, the steering signal in each task-pure local cluster is low-rank, and different query tasks benefit from emphasizing different families of local-basis directions under the shared routing protocol. To address this, we propose MAST (Medoid-Anchored Subspace Steering), a training-free, input-adaptive activation steering framework. MAST groups samples by task, performs multi-center clustering within each group, and selects one representative medoid per cluster. At inference time, each query is routed to multiple representative medoids. For each matched cluster, a low-rank local steering direction is extracted and aggregated across clusters to produce the final steering vector. Building on MAST, MAST-PR (Probe-Reweighted) uses a lightweight task-aware probe to reweight task-specialized local-direction families, yielding a further average improvement of +8.09 points over MAST.

Figures & tables

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Steer Like the LLM: Activation Steering that Mimics Prompting

    May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model SteeringLLM Prompting

  2. ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing

    Mar 25, 2026Yaopei Zeng, Congchao Wang, Blake JianHang Chen +1Multimodal Large Language ModelsLLM Routing