Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding

Amirmohammad Izadi; Hosein Hasani; Fatemeh Askari; Mobin Bagherian; Sadegh Mohammadian; Mohammad Izadi; Mahdieh Soleymani Baghshah

cs.CVSep 28, 2025

Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding

Authors: Amirmohammad Izadi, Hosein Hasani, Fatemeh Askari, Mobin Bagherian, Sadegh Mohammadian, Mohammad Izadi, Mahdieh Soleymani Baghshah

Abstract

Large vision-language models (LVLMs) perform well on multimodal tasks, but their ability to reason and precisely align visual and textual information still has room for improvement. In this study, we show that external visual cues, such as symbols or grid lines, help LVLMs form more accurate connections between visual components, such as objects, and their corresponding textual descriptions, improving their grounding and reasoning abilities. We introduce the concept of Grounding IDs, which are latent identifiers that arise within the model as a result of external cues structuring both visual and textual modalities. Our analysis reveals that partition-inducing external cues lead to Grounding IDs that make better alignment between corresponding visual and text representations, helping the model focus on relevant information. We find that Grounding IDs enhance attention between related components, improving cross-modal grounding and reducing hallucinations. Overall, our results show that Grounding IDs are a key mechanism that enables external cues to improve cross-modal alignment, reduce errors, and enhance the overall performance of LVLMs across a range of multimodal tasks.

Figures & tables

Figure 1: Conceptual overview of Grounding IDs . An input image is augmented with simple visual cues (e.g., {@, #, $}) and paired with a prompt that explicitly includes these symbols. Embeddings with the same Grounding IDs are displayed in matching colors across modalities, illustrating the reinforced binding between partitions and their corresponding textual descriptions.

Panel of Figure 2 — Figure 2: Illustration of attention patterns under baseline and structured inputs in the scene description task. (a) One dataset sample (top: baseline, bottom: structured). (b) Within-modality visual attention matrices. (c) Cross-modality attention matrices. Values are averaged over 500 samples and layers 22–27 of Qwen2.5-VL.

Panel of Figure 3 — Figure 3: Analysis of the modality gap. (a) Cross-modal alignment across layers, showing that improvements emerge in layers 22–27. (b) Average alignment in layers 22–27, reported separately for four partitions. Object embeddings under structured inputs achieve higher alignment than the baseline (dashed line), and symbol embeddings achieve even stronger alignment than objects.

Panel of Figure 4 — Figure 4: Activation swap experiment. (a) Procedure in a case where source ( $c^{\prime}$ ) and target ( $c$ ) contain the same objects. Activations from the & and @ partitions of $c^{\prime}$ are patched into $c$ , producing the patched context $c^{*}$ . Predictions in $c^{*}$ follow the transferred bindings (gray) rather than host symbols. (b) Average log probabilities of $c$ and $c^{*}$ over valid row–symbol–object combinations. Rows and columns indicate the two selected query symbols and their corresponding objects.

Panel of Figure 5 — Figure 5: Causal mediation analysis across layers. (a) Average logit differences between $\mathbf{o}^{s}_{\sim s}$ and $\mathbf{o}^{\sim s}_{s}$ across layers, showing where the model begins to favor the bound object $\mathbf{o}^{s}_{\sim s}$ . (b) Signal-to-noise scores for attention differences between $\mathbf{o}^{s}_{\sim s}$ and $\mathbf{o}^{\sim s}_{s}$ patches across heads and layers.

Panel of Figure 6 — Figure 6: Characteristics of Grounding IDs. (a) Cosine similarity between averaged differential vectors of symbol patches (vertical) and their corresponding Grounding IDs (horizontal). (b) Disjoint-symbol control for the activation-swap test. The source context contains the symbols {&, $,#,@}, while the target context contains a nonoverlapping set {!,%,$ \times$ , +}.

Figure 7: Averaged cross-attention behavior on the synthetic dataset with 20 objects. Attention is computed using a max operation with a window size of 5 and stride of 1 across generated tokens.

# Obj.	Method	Precision	Recall	F1	Acc.
10	Baseline	0.56	0.56	0.58	0.42
	Structured (text-only)	0.59	0.68	0.63	0.46
	Structured (img-only)	0.53	0.59	0.56	0.38
	Structured (both)	0.74	0.58	0.65	0.48
15	Baseline	0.30	0.49	0.37	0.24
	Structured (text-only)	0.33	0.61	0.44	0.27

Table 1: Evaluation of the scene description task on synthetic datasets, each containing 500 samples with 10, 15, or 20 unique objects per image.

Figure 8: Cross-attention behavior on the MS-COCO dataset across generated tokens, computed using a max operation with a window size of 10 and stride of 1.

Model	Method	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Inf. Time(s)
LLaVA-1.5	Baseline	51.60	13.20	3.41
	OPERA	48.00	13.52	20.91
	VCD	54.40	14.28	7.81
	SPARC	55.20	12.78	4.50
	Structured	41.00	12.04	3.94
Qwen2.5-VL	Baseline	32.40	7.97	3.31

Table 2: Evaluation on 500 MS-COCO samples using CHAIR metrics across open- and closed-source models. Results are reported for sentence-level (CHAIR s ) and instance-level (CHAIR i ) hallucination rates.

Counting Accuracy
Model	Baseline	VISER	Grounding IDs
Qwen2.5-VL (3B)	30.00	37.83	43.00
Qwen2.5-VL (7B)	29.67	43.33	53.00
GPT-4o	10.50	26.50	32.33
Visual Search Accuracy
Model	Baseline	VISER	Grounding IDs

Table 3: Performance on counting and visual search benchmarks.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Mean cross-attention maps (layers 14–28) from textual tokens to image rows. Structural cues amplify the model’s inherent top-to-bottom, left-to-right traversal bias and produce sharper row-aligned attention patterns.

Figure 10: Examples of per-object traversal heatmaps for a structured image. Each panel shows the attention distribution for a specific shape–color pair. Structural cues guide the model toward sharper localization and promote systematic top-to-bottom and left-to-right traversal.

Figure 11: Examples of per-object traversal heatmaps for the baseline. Each panel shows the attention distribution for a specific shape–color pair. The baseline also exhibits a general top-to-bottom and left-to-right pattern, but accuracy degrades over generation.

Figure 12: ICG scores across layers for baseline vs. structured inputs. Structured inputs consistently yield higher ICG, reflecting stronger row-level grouping.

Figure 13: Proximity-controlled logit-lens analysis. The two panels test whether symbol propagation can be explained only by local adjacency to the nearest symbol. In both layouts, visual tokens are mainly associated with their partition symbol rather than simply the closest symbol, arguing against a purely local semantic-blurring explanation.

Variant	Precision	Recall	F1	Accuracy
Baseline	0.14	0.45	0.21	0.12
Numbers (no line)	0.53	0.43	0.47	0.31
Letters + Line	0.64	0.48	0.55	0.38
Numbers + Line	0.56	0.58	0.57	0.40
Symbols + Line	0.65	0.59	0.62	0.40
Grid 4 $\times$ 4 Numbered	0.33	0.40	0.36	0.22

Appendix

Table 4: Performance metrics (precision, recall, F1, accuracy) on the scene description task across synthetic variants with 20 objects. Explicit structural cues (such as lines, labels, or symbols) consistently improve performance compared to the baseline.

Figure 14: Qualitative example comparing captions generated with and without structured scaffolding. The figure shows the input image, the applied scaffolding structure, the caption produced by our method, and the corresponding baseline caption. Hallucinated objects in the baseline output are highlighted in red .

Model	POPE	Method	Accuracy	Precision	Recall	F1 Score	Yes (%)
LLaVA-1.5	Random	Base	87.27	97.47	76.66	85.82	39.53
		OPERA	86.17	97.00	74.88	84.52	38.93
		VCD	85.43	93.86	75.99	83.99	40.70
		SPARC	85.10	98.90	71.34	82.89	36.50
		Structured	89.23	95.19	82.75	88.53	43.67
	Popular	Base	85.93	94.28	76.63	84.54	40.80

Appendix

Table 5: Results of POPE evaluation on the MSCOCO dataset across Random, Popular, and Adversarial splits. Metrics reported are Accuracy, Precision, Recall, F1 Score, and Yes (%) for LLaVA-1.5.

Model	POPE	Method	Accuracy	Precision	Recall	F1 Score	Yes (%)
Qwen2.5-VL	Random	Base	89.73	98.39	80.91	88.80	41.37
		OPERA	88.83	98.76	78.85	87.69	40.27
		VCD	89.90	98.55	81.13	88.99	41.43
		SPARC	85.33	95.91	71.76	82.10	35.07
		Structured	85.67	98.22	72.90	83.69	37.43
	Popular	Base	88.83	96.07	81.09	87.95	42.40

Appendix

Table 6: Results of POPE evaluation on the MSCOCO dataset across Random, Popular, and Adversarial splits. Metrics reported are Accuracy, Precision, Recall, F1 Score, and Yes (%) for Qwen2.5-VL.

Method	Subset	Accuracy	Precision	Recall	F1	Yes(%)
Ensemble	Adversarial	85.72	93.96	76.59	84.40	41.08
	Random	87.82	99.06	76.48	86.31	38.78
	Popular	86.82	96.41	76.66	85.41	40.00
Structured	Adversarial	83.73	93.60	72.65	81.80	39.08
	Random	85.67	98.22	72.90	83.69	37.42
	Popular	85.07	96.92	72.72	83.09	37.86

Appendix

Table 7: POPE results for Qwen2.5-VL using multi-scale structured cues with majority voting.

Method	Wild	Hom.	Het.
Baseline	63.13	76.10	43.49
Grounding IDs	66.63	78.60	49.88

Appendix

Table 8: Results on the ROPE benchmark ( Zhou et al., 2024 ) .

Figure 15: Average log probabilities in the multi-object-per-row activation-swapping setting, reported separately for color and shape. In the standard setting, the highest probabilities align with the original symbol–object associations. After activation swapping, the probability mass shifts toward the transferred associations for both attributes.

Panel of Figure 16 — Figure 16: Attention patterns of InternVL3.5 (8B) under baseline (top) and structured (bottom) inputs for the scene description task. (a) Within-modality visual attention. (b) Cross-modality attention.

Panel of Figure 17 — Figure 17: Attention patterns of Qwen2.5-VL (3B) under baseline (top) and structured (bottom) inputs for the scene description task. (a) Within-modality visual attention. (b) Cross-modality attention.

Panel of Figure 18 — Figure 18: Average log probabilities of $c$ and $c^{*}$ over valid row–symbol–object combinations for InternVL3.5 (a) and Qwen2.5-VL 3B (b). Rows and columns correspond to the selected query symbols and their associated objects.

Model	Method	CHAIR ${}_{s}\downarrow$	CHAIR ${}_{i}\downarrow$
Qwen2.5-VL 3B	Baseline	32.40	8.82
Qwen2.5-VL 3B	Structured	24.40	7.32
InternVL3.5 8B	Baseline	32.40	6.09
InternVL3.5 8B	Structured	26.40	8.08

Appendix

Table 9: CHAIR results on 500 MS-COCO samples for Qwen2.5-VL (3B) and InternVL3.5 (8B).

Multimodal Grounding

Multimodal grounding links language and other signals to the specific objects, regions, events, or actions that support an interpretation.

See all 185 papers

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
Oct 6, 2026Seulgi Kim, Zhixiong Zhang, Xinwei Zhang +2Vision-Language Grounding Representation Geometry in VLMs
Georgia Institute of Technology · Work done as an intern at Amazon. · Amazon
Seeing to Generalize: How Visual Data Corrects Binding Shortcuts
Feb 16, 2026Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza +3Cross-Modal Learning Shortcut Learning
Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.

Feb 25, 2026J/K move · Enter open · S save

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Yongchang Zhang, Oliver Ma, Tianyi Liu, Guangquan Zhou+1

Southeast University · AIIA, Ministry of Education, China · University of Oxford

Multimodal Reasoning VLM Reasoning VLM Hallucination Mitigation Object Hallucination in VLMs

Open paper AskPDF arXiv

Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.

Abstract

Multimodal Grounding

Explore similar work

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts