From What to Which: Decoding Modifier Grounding in Frozen MLLMs

Barbara Toniella Corradini; Caterina Gallegati; Ludovica Genovese; Vittorio Murino

cs.CVOct 8, 2026

From What to Which: Decoding Modifier Grounding in Frozen MLLMs

Authors: Barbara Toniella Corradini, Caterina Gallegati, Ludovica Genovese, Vittorio Murino

Organizations: AI for Good · AI for Good (AIGO), Istituto Italiano di Tecnologia, Italy · University of Siena, Italy · University of Genoa, Italy · University of Verona, Italy

Abstract

As Multimodal Large Language Models (MLLMs) can describe increasingly complex visual scenes, token-level grounding becomes crucial. Yet, when an MLLM generates "the yellow banana on the left", established grounding approaches focus on what is in the image ("banana"), overlooking tokens that help describe which instance is meant ("yellow", "left"). In this work, we ask whether frozen MLLM representations contain decodable grounding information about the referred instance across generated tokens, extending to modifiers such as attributes, spatial expressions, and relational/action terms. To address this question, we introduce OTTER, a lightweight supervised probe over frozen MLLM representations that uses Optimal Transport (OT) to align generated tokens with visual regions and produce compact grounding maps. Our results show that (i) instance-discriminative visual information can be decoded from modifier tokens, with the clearest evidence for spatial terms, but (ii) is not confined to them, as contextualized object nouns also carry referential information; (iii) the recovered grounding remains informative under context perturbations, while selected regions remain relevant to generation; and (iv) the learned OT-based grounding extends beyond the controlled setting to free generation and cross-dataset transfer.

Figures & tables

Figure 1: Method Overview. The frozen MLLM produces token and visual patch states, which the trainable OTTER probe projects into a shared space to build a token–patch cost matrix. An entropy-regularized OT solver aligns tokens and visual patches, yielding a token-level grounding map.

Table 2

Table 3

Token group	Scrambled	Isolated
Nouns	$1.000\,[0.99,1.01]$	$1.000\,[0.98,1.02]$
Spatial	$0.982\,[0.97,0.99]$	$\mathbf{1.077\,[1.05,1.10]}$
Attribute	$0.942\,[0.93,0.96]$	$0.914\,[0.87,0.96]$
Function	$0.939\,[0.93,0.95]$	$0.597\,[0.57,0.63]$

Table 5: Context dependence and matched probe controls. (a) normalized Energy-in-GT retention under context perturbation; (b) matched projection controls on RefCOCO-val under the same OT/RIS training.

Table 5

Panel of Figure 2 — Figure 2: Qualitative comparison on token-level grounding. We compare OTTER against TAM, attention, semantic similarity, center, and random baselines.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone (layer)	Diag. IoU	mIoU	[email protected]
Qwen2.5-VL-3B L9	28.75	27.51	14.14
Qwen2.5-VL-3B L18	38.45	35.23	29.69
Qwen2.5-VL-3B L27	44.22	41.06	40.92
Qwen2.5-VL-3B L32	41.39	38.20	34.67
Qwen2.5-VL-3B L35	28.76	28.56	11.88
Qwen2.5-VL-3B L36	27.30	27.10	9.65

Appendix

Table 8: Cross-backbone layer screening. Metrics are percentages. The selection score is the area-matched diagnostic IoU; calibrated mIoU is reported as the task metric. For each backbone, the selected layer is highlighted in bold .

Backbone (layer)	Noun	Attribute	Spatial	Rel./act.	Token avg.
Qwen2.5-VL-3B L27	37.91	39.47	39.05	32.75	37.30
Qwen3-VL-2B L14	29.95	36.82	31.51	32.34	32.66
InternVL3.5-2B L14	12.19	12.29	12.82	10.23	11.88
Gemma-3-4B L17	12.08	10.93	10.48	9.45	10.74

Appendix

Table 9: Mini token-type grounding check on RefCOCO modifier expressions using sweep heads. Rows report per-token maps grouped by token type.

	mIoU at Training Scale (Images)							Full Metrics at 2500 Images
Variant	50	100	300	600	900	1500	2000	mIoU	cIoU	[email protected]	Diag. IoU
softmax_semantic_only	21.83	23.81	31.31	37.51	40.36	42.09	42.96	43.74	42.81	45.79	48.65
softmax_no_ot	22.08	24.76	29.42	38.17	40.36	41.87	42.17	43.74	43.19	45.52	49.15
softmax_fixed_full	22.24	24.83	29.04	38.76	40.44	42.36	41.84	43.69	42.58	45.42	50.48
ot_semantic_only	21.02	24.53	34.47	39.69	40.50	43.26	44.74	45.66	46.08	49.74	49.57
ot_sem_spatial	20.84	24.11	34.12	39.63	41.70	42.92	44.63	45.12	45.41	47.97	49.67

Appendix

Table 10: Combined architecture and training-scale ablation at Qwen2.5-VL-3B layer 27 on RefCOCO validation. The left side reports the mIoU across increasing numbers of requested training images. The right side reports detailed metrics (mIoU, cIoU, [email protected], Diag. IoU) for all configurations at the largest training scale (2500 images). The overall best configuration is highlighted in bold .

Cost terms exploited: gain from adding attention+spatial to the semantic cost
	Token group	$\Delta$ Soft-IoU	$\Delta$ gap	$\Delta$ peak/mean
Effect of OT at full cost ( ot_full $-$ sm_no_ot )	Spatial	$+0.021\,[.019,.023]$	$-0.018\,[\text{-}.022,\text{-}.013]$	$\mathbf{+5.41\,[5.21,5.62]}$
	Attribute	$+0.018\,[.015,.021]$	$-0.028\,[\text{-}.033,\text{-}.022]$	$\mathbf{+5.93\,[5.65,6.22]}$
	Noun	$+0.026\,[.025,.027]$	—	$\mathbf{+3.85\,[3.75,3.96]}$
OT uses them ( ot_full $-$ ot_sem )	Spatial	$+0.024\,[.022,.026]$	$+0.031\,[.027,.036]$	$+0.77\,[.64,.90]$
OT uses them ( ot_full $-$ ot_sem )	Attribute	$+0.022\,[.019,.026]$	$+0.030\,[.023,.036]$	$-1.32\,[\text{-}1.59,\text{-}1.06]$

Appendix

Table 11: Controlled operator ablation with

95\%

paired bootstrap CIs (

n

per group). Concentration is reported as peak-to-mean ratio; higher is more concentrated. This non-saturating metric reveals the operator effect that normalized entropy hides. Effect of OT at full cost isolates the Sinkhorn operator; cost terms exploited shows the gain from adding attention+spatial to each operator. All intervals shown exclude zero. Gap is undefined for nouns because there are no same-category instance distractors.

Explore similar work

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
Jul 7, 2026Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen +1Multimodal Grounding LLM Uncertainty Estimation
Amazon
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Jul 27, 2026Tianyi Gao, Han Fang, Tianyi Ding +9Multimodal Grounding Multimodal Large Language Models
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · Beijing University of Posts and Telecommunications +2
Faithful Grounded Visual Reasoning via Learned Proxy-Tokens
Jun 22, 2026Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo +1Visual Question Answering Visual Reasoning
Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Jul 7, 2026, cs.CVJ/K move · Enter open · S save

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen, Tal Remez

Amazon

Multimodal Grounding LLM Uncertainty Estimation Multimodal Model Evaluation VLM Hallucination Mitigation

Open paper AskPDF arXiv

Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.

Abstract

Explore similar work

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding

Faithful Grounded Visual Reasoning via Learned Proxy-Tokens