MAST: Medoid-Anchored Subspace Steering for Visual Grounding in Multimodal LLMs
Authors: Zhitao Chen, Zhenxin Cai, Jingyuan Li, Jin Sha
Organizations: School of Electronic Science and Engineering, Nanjing University, China · College of Electronic Science and Technology, National University of Defense Technology, China
Existing activation steering methods for multimodal large language models (MLLMs) use the global activation mean as the steering vector, which tends to cause cross-task interference and unstable performance. We trace this failure to the multi-centered geometry of task activations. The global mean lands in the empty region between clusters and is faithful to no single task. Moreover, the steering signal in each task-pure local cluster is low-rank, and different query tasks benefit from emphasizing different families of local-basis directions under the shared routing protocol. To address this, we propose MAST (Medoid-Anchored Subspace Steering), a training-free, input-adaptive activation steering framework. MAST groups samples by task, performs multi-center clustering within each group, and selects one representative medoid per cluster. At inference time, each query is routed to multiple representative medoids. For each matched cluster, a low-rank local steering direction is extracted and aggregated across clusters to produce the final steering vector. Building on MAST, MAST-PR (Probe-Reweighted) uses a lightweight task-aware probe to reweight task-specialized local-direction families, yielding a further average improvement of +8.09 points over MAST.
Figures & tables
Figure 1: An illustration of the MAST principle. Visual tasks form their own local clusters in the activation space, and the global mean (the red cross × ) lands in the empty region between clusters. MAST routes a query (the ⋆ ) to the nearest medoid and, within its local cluster, represents the local steering signal with low-rank local subspace directions. MAST-PR can reweight these directions using task-specialized weights.
Figure 2
Figure 4: Overview of MAST . Step 1 (offline). Within each task, K -medoids clustering ( K=5 ) builds the prototype bank B of 20 medoids, whose directions are largely misaligned with the task global mean. Step 2 (online). Adaptive routing by sq,c selects the top- k medoids ( k=3 ). Step 3 (online). Each selected cluster is blended into vc and summed into the steering vector v . Contrastive directions are extracted from teacher-forced support completions, whereas routing uses separately extracted answer-free prompt contexts (Section 4.1 ). The extension MAST-PR reweights the task-specialized directions βτ via a lightweight probe.
LLaVA-v1.5
Qwen2-VL
Method
Relation
Count
Distance
Depth
Average
Relation
Count
Distance
Depth
Average
No-steering
59.62
47.46
23.33
73.33
50.94
78.08
67.14
77.71
86.67
77.40
Norm-Rnd
59.42
49.52
27.08
73.33
52.34
77.31
66.51
77.29
87.08
77.05
CAA
56.15
46.67
22.92
73.12
49.72
78.46
67.14
77.71
86.25
77.39
L2S
61.54
47.94
19.58
72.92
50.49
78.27
67.14
77.08
86.46
77.24
Act-Add
57.31
45.56
25.83
73.54
50.56
78.85
66.98
78.12
85.83
77.45
Table 1: Accuracy ( % ) on the four CV-Bench visual grounding tasks (test split), each model at the intervention layer chosen by its own layer scan. Best per column in bold and second best underlined . * Norm-Rnd is a norm-matched random-direction control, and Task-Oracle applies the optimal hard direction weights of the ground-truth task as an upper bound in the four-task-average sense, excluded from the ranking. Baseline paradigms and frozen hyperparameters are described in Sections 5.1 and 5.2 . Baseline references are CAA ( Rimsky et al., 2024 ) , L2S ( Parekh et al., 2025 ) , Act-Add ( Turner et al., 2024 ) , Linear-AcT ( Rodriguez et al., 2025 ) , ACT ( Wang et al., 2025 ) , and RepE ( Zou et al., 2025 ) .
Figure 5Table 6
Figure 6: CV-Bench accuracy gain of MAST over No-steering at every language-model (LM) layer of LLaVA-v1.5 (validation split). Layer 16 (black box) is the default intervention layer.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Scope
Task
Clusters
Separability ↑
DB ↓
Top-1
Per-task
Relation
5
0.192
1.769
100.00
Per-task
Count
5
0.144
2.051
100.00
Per-task
Distance
5
0.159
2.142
100.00
Per-task
Depth
5
0.170
1.978
100.00
All
all (4 tasks)
20
0.150
2.035
76.69
Appendix
Table S1: Clustering quality of the label-grouped K -medoids clusters. Separability (higher is better), DB is the Davies-Bouldin index (lower is better), and Top-1 is the top-1 routed same-task accuracy. Per-task routing is perfect, and the combined 20-cluster setting stays far above the 20% random baseline.
Figure S1: Full four-task UMAP projection of CV-Bench support activations. Support samples are colored by cluster, stars mark the cluster medoids, and the black cross marks the global mean. Across all four tasks the global mean lands in the empty region between clusters, confirming that the multi-center geometry is a general phenomenon.
Figure S2: Full four-task cumulative variance ratio of the singular value decomposition of the intra-cluster steering vectors. Solid lines denote per-cluster curves, the black dashed line the global baseline without clustering, and the dotted grey line the 90% threshold. Across all tasks the leading singular directions capture at least 90% of the variance well below the ambient dimension, confirming that the intra-cluster low-rank structure is a general property.
Task
C0
C1
C2
C3
C4
Average
Relation
0.839
0.750
0.573
-0.106
0.823
0.576
Count
0.895
0.063
0.812
0.696
0.802
0.654
Distance
0.387
0.337
0.491
0.407
0.466
0.418
Depth
0.518
0.243
0.364
0.439
0.254
0.364
Appendix
Table S2: Cosine similarity between each cluster’s steering direction and the global mean direction, per CV-Bench task. The final column averages over the five clusters. The global mean stays far from every local cluster direction, confirming that it faithfully represents none of them.
Figure S3: Full four-task pairwise cosine similarity between the five cluster medoid steering directions within each CV-Bench task. Red denotes negative correlation and blue positive. The geometric tasks Distance and Depth exhibit the most severe near-orthogonal conflict among cluster directions, while Relation and Count show a less extreme but still substantial directional spread.
Figure S4: Average intra-task pairwise cosine similarity between cluster steering directions across the four CV-Bench tasks. Lower means more severe directional conflict, lowest for Depth and Distance.
True \ Pred.
Relation
Count
Distance
Depth
Recall
Relation
63.54
5.09
20.16
11.20
100.00
Count
4.14
92.84
1.61
1.41
100.00
Distance
27.26
6.05
58.88
7.81
100.00
Depth
11.62
2.26
49.62
36.50
18.67
Appendix
Table S3: Soft routing-probability matrix of the MAST-PR task probe on the CV-Bench test split (LLaVA-v1.5, layer 16). Row i , column j is the mean probability assigned to task j over queries whose true task is i , diagonal entries (bold) the mass kept on the correct task, and the rightmost column auxiliary top-1 task-classification accuracy.
Posterior noise ϵ
Relation
Count
Distance
Depth
Average
0.00
87.69
74.60
51.67
90.42
76.09
0.25
92.69
68.41
47.29
96.88
76.32
0.50
95.00
59.05
37.08
97.92
72.26
0.75
94.23
52.38
35.21
98.75
70.14
1.00
91.54
47.94
30.63
98.12
67.06
Equal-weight MAST
79.04
53.81
39.58
99.58
68.00
Appendix
Table S4: Uniform task-posterior perturbation of MAST-PR on the LLaVA-v1.5 CV-Bench test split. All rows retain identical medoid routing and steering assets; only the task-probe posterior used for direction reweighting changes.
Depth weighting rule
Accuracy
Equal-weight MAST
99.58
Native soft MAST-PR
90.42
Force Distance-specialized weights
12.08
Appendix
Table S5: Counterfactual intervention on Depth (LLaVA-v1.5). Relative to native soft MAST-PR, forcing Distance-specialized weights lowers accuracy by 78.34 points while leaving medoid routing unchanged.
Method
Relation
Count
Distance
Depth
Average
Equal-weight MAST
79.04
53.81
39.58
99.58
68.00
Native soft MAST-PR
87.69
74.60
51.67
90.42
76.09
MAST-PR-Safe (fixed t=0.60 )
84.23
72.54
38.96
99.58
73.83
Appendix
Table S6: Fixed confidence-gated MAST-PR safety policy on LLaVA-v1.5 CV-Bench. The threshold is a pre-specified conservative policy, not a test-set-selected hyperparameter.
Model
Relation
Count
Distance
Depth
LLaVA-v1.5
79.04
53.81
39.58
99.58
Qwen2-VL
81.15
67.62
82.71
91.88
Appendix
Table S7: Cross-seed stability of MAST on CV-Bench (test split). Each entry is the mean per-task accuracy ( % ) of MAST over three K -medoids clustering seeds ( 0 , 42 , 123 ). The standard deviation across seeds is 0.00 in every cell on both models, so the means coincide exactly with the single-run MAST accuracies in the main-paper CV-Bench table.
Method
K
C
Offline (s)
Online ( μ s/q.)
Bank (MiB)
Global mean
–
1
–
–
0.016
MAST
1
4
0.281
0.72
0.188
MAST
5
20
3.930
0.65
0.938
MAST
10
40
4.071
0.68
1.875
MAST
20
80
4.703
0.66
3.750
MAST-PR
5
20
shared
–
3.438
Appendix
Table S8: Incremental efficiency cost on LLaVA-v1.5 CV-Bench support and test contexts in fp32. K is medoids per task and C=4K the total prototype count. “Online” is the base-MAST routing-and-aggregation vector kernel, excluding common MLLM generation; base MAST precomputes its local reconstructions and thus performs no online SVD. MAST-PR shares the same offline bank construction and retains additional bases for query-dependent reweighting; its vector-kernel latency is not included in the base-MAST microbenchmark.
Method
Relation
Count
Distance
Depth
Average
Positive-instruction no-steering
56.15
43.04
33.33
56.67
47.30
Causal-aligned MAST
50.77
16.46
9.17
11.67
22.01
Appendix
Table S9: Answer-free causal-alignment control on the held-out CV-Bench validation split with LLaVA-v1.5 at layer 16. Positive and negative support states differ only in their behavior instruction and are both read at the final prompt token; routing and query contexts use the positive instruction at that same position. Accuracy ( % ).
Task
Hardest baseline
b
c
McNemar p
LLaVA-v1.5
Relation
L2S
129
38
<10−3
Count
Norm-Rnd
49
22
2.0×10−3
Distance
Linear-AcT
122
72
<10−3
Depth
RepE
119
2
<10−3
Qwen2-VL
Appendix
Table S10: McNemar’s paired significance test of MAST against the steering baselines on CV-Bench (test split). For each model and task we report the least favorable (largest p ) of the seven steering baselines, and b and c are the discordant-pair counts ( b>c favors MAST).
Figure S5: Qualitative examples on the four CV-Bench tasks, one per task. They illustrate cases in which MAST uses a routed local steering signal and produces the correct answer where the displayed No-steering output is incorrect.
Variant
Md
Mc
LB
Sh
Relation
Count
Distance
Depth
Average
MAST-PR
✓
✓
✓
✓
87.69
74.60
51.67
90.42
76.09
MAST
✓
✓
✓
✓
79.04
53.81
39.58
99.58
68.00
w/o Shrinkage
✓
✓
✓
×
78.46
53.49
36.46
99.58
67.00
w/o Local Basis
✓
✓
×
✓
79.42
53.33
38.54
99.38
67.67
w/o Multi-center
✓
×
✓
✓
49.42
43.33
14.17
51.25
39.54
w/o Medoid
×
×
×
×
50.19
41.90
25.42
71.88
47.35
Appendix
Table S11: Full per-task accuracy ( % ) of the leave-one-out component ablation on CV-Bench (test split). Md is medoid prototype, Mc multi-center routing, LB local basis, and Sh adaptive shrinkage. The MAST block removes one component at a time under equal weighting.
Subspace representation
Local
Data-driven
Task-cond.
Relation
Count
Distance
Depth
Average
Random local basis
✓
×
✓
80.05±0.44
52.88±0.26
40.72±0.82
99.17±0.00
68.20
Raw local directions
✓
✓
✓
90.31
55.08
55.67
98.83
74.97
Global PCA subspace
×
✓
✓
90.46
54.57
66.33
99.33
77.67
Local PCA/SVD subspace
✓
✓
✓
90.62
72.08
74.17
99.33
84.05
Appendix
Table S12: Full controlled task-oracle comparison on LLaVA-v1.5 CV-Bench. All rows share the MAST backbone, medoid routing, K=5 , r=8 , layer, support protocol, and evaluation. For task-conditioned rows, ground-truth task labels are used only to select a subspace direction and weight on a disjoint validation split, which are frozen before testing.
Figure S6: Sensitivity of the shrinkage bound αmax (a) and the local basis rank r (b) on CV-Bench (validation split). Thin lines are per-task accuracy and the thick line is the four-task average. Sweeping αmax over its full range changes the average by only 1.40 points and sweeping r from 1 to 4096 by only 0.76 points, confirming that the effective steering signal concentrates in the leading few singular directions and that MAST is insensitive to both.
Param
Value
Relation
Count
Distance
Depth
Average
K
1
50.77
38.61
10.00
44.17
35.89
2
50.77
44.30
18.33
90.83
51.06
4
74.62
49.37
27.50
99.17
62.66
5
79.23
50.00
43.33
98.33
67.72
8
74.62
54.43
35.83
100.00
66.22
16
95.38
61.39
33.33
100.00
72.53
Appendix
Table S13: Full one-at-a-time hyperparameter sweep on CV-Bench (validation split). Each block varies one parameter with the others at their defaults (bold). K is the number of medoids, r the local basis rank, k the routing size, and αmax the upper shrinkage bound. The average is nearly flat in r and αmax , confirming that MAST is insensitive to both. For K and k the average is maximized at the extremes ( K=64 , k=1 ), but only because Relation and Count saturate: the geometric task Distance instead collapses there ( 20.83 at K=64 , 26.67 at k=1 ), so the moderate defaults K=5 and k=3 are chosen to preserve Distance rather than the average-maximizing corner.
Large Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and attention to global contexts, followed by representation editing during inference time. Experimental results on multiple datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git.
Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.
Routing balances performance and cost in hybrid systems by escalating selected queries from a lightweight model to a powerful but expensive one. Hidden-state probes provide an effective routing signal by predicting the small model's correctness from hidden states it already computes. Although effective in text-only settings, their behavior on multimodal LLMs (MLLMs) is less understood. We find that correctness is harder to predict from hidden states when questions are paired with images rather than with captions of those images. Two training-free dependence measures, HSIC and label CKA, show the same pattern. We first investigate whether changing hidden states used by the probe improves correctness prediction for MLLMs. We propose Attention Probe, which learns an attention query to pool frozen token states into a routing feature and improves over the standard probe. However, pooling only recombines hidden states computed for answer generation, not for deciding whether the answer should be trusted. We therefore build a decision layer inside the existing MLLM and train it for the routing decision. The resulting router, ReLope (KL-Regularized LoRA Probe), adapts a separate copy of one MLLM layer with LoRA and regularizes it with a KL-penalized stochastic bottleneck. This decision branch shares the frozen lower layers and leaves the model's answer-generation path unchanged. ReLope attains the highest correctness prediction AUC on all five benchmarks and three backbones we evaluate and improves the accuracy versus cost trade-off at negligible overhead. These results suggest a new perspective and a practical technical path on routing: rather than reading decisions off states built for generation, routing decisions can be learned by a dedicated decision layer inside the model. Code: https://github.com/Spinozaaa/ReLope