Recognizing Dialogue Ambiguity via Candidate Set Visual Grounding

Shun Inadumi; Lee Sangmyeong; Koichiro Yoshino

cs.CLOct 14, 2026

Recognizing Dialogue Ambiguity via Candidate Set Visual Grounding

Authors: Shun Inadumi, Lee Sangmyeong, Koichiro Yoshino

Organizations: The University of Osaka · Guardian Robot Project, RIKEN · Nara Institute of Science and Technology

Abstract

Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects in a visual scene, it is necessary to maintain and update the set of plausible referents as context accumulates, without assuming that the available information identifies a unique object. However, prior work on interactive visual grounding, which localizes objects referred to by linguistic expressions, has focused on identifying a unique object once sufficient dialogue context becomes available. To address this gap, we propose Candidate Set Visual Grounding (CSVG), a task for estimating the set of objects consistent with the given dialogue context. We further introduce CandVG, a benchmark that evaluates how the candidate set changes across dialogue turns. Our evaluation of vision-language models shows that stronger visual grounding capability does not necessarily yield better CSVG performance. In particular, models have difficulty predicting the complete candidate set, owing to challenges in candidate generation and premature commitment to a small set of candidates in early turns. Nevertheless, we find that this limitation can be partially mitigated in the evaluated large-scale models using instructions designed for CSVG.

Figures & tables

Figure 1: Concept of CandVG: Checkboxes indicate objects that remain plausible referents of the initial ambiguous utterance under the given dialogue context. Each accumulated turn yields a separate evaluation sample.

Panel of Figure 2 — Figure 2: Distributions of candidate set size per sample and number of unique candidates across each dialogue

Pair	Acc	P	R	F1
Pair1	0.962	0.959	0.944	0.943
Pair2	0.971	0.966	0.924	0.935

Table 1: Annotation agreement for two annotator pairs

Figure 4: F1 scores on CSVG with and without ground-truth candidate sets, where 27B+ denotes the 27B–38B variants.

Panel of Figure 5 — Figure 5: Transitions of candidate sets across dialogue turns, showing the average numbers of ground-truth, predicted, and correctly predicted BBoxes. The analysis tracks the same 1,234 dialogues across the first three turns, covering both CSVG and CVG cases.

Figure 6: Examples of candidate set updates across turns: Green BBoxes indicate ground-truth candidates across both CSVG and CVG cases. The symbols $\spadesuit$ , $\blacklozenge$ , and $\clubsuit$ denote objects predicted by Qwen3 VL 4B , Qwen3.5 27B , and GPT-5.6 Luna , respectively.

Table 3: Improvements and degradations in CSVG precision and recall from the base prompt after adding Task-Aware Instructions (TAI). Colored values indicate improvements of at least 0.03 over the base prompt.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Annotation interface used for candidate set collection

Dialogues	1,249
Samples	4,833
Average of Turns	3.869
Average of Objects	2.355
Average of Unique Objects	4.796
Total Words	124,047
Unique Words	2,058

Appendix

Table 4: Statistics on CandVG

Figure 8: Transition of candidate set sizes across turns

Panel of Figure 9 — Figure 9: Heatmaps of point annotations across turns: Coordinates are normalized by image width and height.

Models	HuggingFace ID or API Name
Intern3.5 VL 2B	OpenGVLab/InternVL3_5-2B-HF
Intern3.5 VL 4B	OpenGVLab/InternVL3_5-4B-HF
Intern3.5 VL 8B	OpenGVLab/InternVL3_5-8B-HF
Intern3.5 VL 38B	OpenGVLab/InternVL3_5-38B-HF
Qwen3 VL 2B	Qwen/Qwen3-VL-2B-Instruct
Qwen3 VL 4B	Qwen/Qwen3-VL-4B-Instruct

Appendix

Table 5: Details of the models for the experiments

Panel of Table 6 — Table 6: Prompts used in the experiments: All models share the same base prompt and differ only in output format. The selection setting optionally appends a ground-truth candidate set.

Models	Acc ( $\uparrow$ )	P ( $\uparrow$ )	R ( $\uparrow$ )	$\geq$ 2 ( $\downarrow$ )
Intern3.5 VL 2B	0.521	0.519	0.533	2.9
Intern3.5 VL 4B	0.568	0.563	0.588	5.1
Intern3.5 VL 8B	0.580	0.575	0.616	11.0
Intern3.5 VL 38B	0.746	0.744	0.748	0.7
Qwen3 VL 2B	0.480	0.478	0.498	3.2
Qwen3 VL 4B	0.622	0.328	0.763	74.2

Appendix

Table 7: Results on the CVG task using [email protected] ( Acc ), precision ( P ), recall ( R ), and the percentage of samples with two or more predicted BBoxes (

\geq

2 [

\%

] )

Models	Turn1	Necessary	Full
Intern3.5 VL 2B	0.265	0.519	0.510
Intern3.5 VL 4B	0.308	0.560	0.560
Intern3.5 VL 8B	0.277	0.581	0.561
Intern3.5 VL 38B	0.297	0.722	0.755
Qwen3 VL 2B	0.172	0.467	0.469
Qwen3 VL 4B	0.285	0.593	0.623

Appendix

Table 8: Results on the CVG task under different dialogue context conditions

Generalist VLMs
Models	CandVG (w/o GT)			CandVG (w/ GT)			gRefCOCO
Models	P	R	F1	P	R	F1	P	R	F1
Intern3.5 2B	0.683	0.230	0.329	0.855	0.319	0.431	0.704	0.324	0.419
Intern3.5 4B	0.715	0.258	0.357	0.895	0.787	0.792	0.798	0.631	0.674
Intern3.5 8B	0.689	0.230	0.331	0.863	0.654	0.661	0.763	0.407	0.496
Intern3.5 38B	0.776	0.268	0.382	0.932	0.696	0.744	0.832	0.578	0.646

Appendix

Table 9: Detailed precision ( P ), recall ( R ), and F1 score on CSVG: w/o GT and w/ GT denote the settings without and with ground-truth candidate sets, respectively.

Generalist VLMs
Models	CandVG (w/o GT)
Models	2 (708)	3 (554)	4 (387)	5 (257)	6 (146)	7+ (232)	All (2,284)
Intern3.5 2B	0.445	0.345	0.293	0.249	0.202	0.161	0.329
Intern3.5 4B	0.451	0.378	0.330	0.292	0.250	0.201	0.357
Intern3.5 8B	0.461	0.350	0.288	0.240	0.192	0.148	0.331
Intern3.5 38B	0.533	0.405	0.330	0.263	0.234	0.174	0.382

Appendix

Table 10: Detailed F1 scores for the CSVG task by candidate set size: Numbers in parentheses indicate the number of samples in each bin.

Generalist VLMs
	Retention [ $\%$ ] (5,424)	Removal [ $\%$ ] (4,677)
Model	Success ( $\uparrow$ )	Success ( $\uparrow$ )	Not Rem. ( $\downarrow$ )	Not Pred. ( $\downarrow$ )
Intern3.5 2B	24.8	9.2	5.5	85.4
Intern3.5 4B	27.3	12.0	4.9	83.1
Intern3.5 8B	27.2	9.6	4.4	86.1
Intern3.5 38B	33.8	12.8	3.4	83.8

Appendix

Table 11: Candidate set retention and removal rates [

\%

] across consecutive dialogue turns: Removal rate outcomes are categorized as Success, Not Removed, and Not Predicted. Numbers in parentheses indicate the total counts of retained and removed candidates.

Visual Grounding

Visual grounding connects language to the image regions, objects, or pixels that support a description, question, or action.

See all 153 papers

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue
May 20, 2026Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan +4Query Reformulation Visually Grounded Reasoning
KTH Royal Institute of Technology, Stockholm, Sweden
When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
Aug 25, 2026Zhengxiang Wang, Owen RambowVLM Evaluation Large Vision-Language Models
Department of Linguistics & IACS Stony Brook University
Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
Jul 30, 2026Lei Yang, Xinze Liu, Dayan Wu +7Hallucination Detection in VLMs LLM Hallucination Mitigation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

May 20, 2026J/K move · Enter open · S save

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue

Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan, Lubos Marcinek+3

KTH Royal Institute of Technology, Stockholm, Sweden

Query Reformulation Visually Grounded Reasoning 3D Visual Grounding Multimodal Understanding Referring Expression Comprehension

Open paper AskPDF arXiv

Grounding language in the physical world requires AI systems to interpret references that emerge dynamically during conversation. While current vision-language models (VLMs) excel at static image tasks, they struggle to resolve ambiguous expressions in spontaneous, multi-turn dialogue. We address this gap by introducing (1) a benchmark for referential communication in dynamic 3D environments, built from 6.7 hours of egocentric VR interaction with synchronized speech, motion, gaze, and 3D scene geometry, and (2) a two-stage grounding pipeline that explicitly resolves conversational ambiguity before visual localization. The benchmark includes over 4,200 manually verified referring expressions spanning full, partitive, and pronominal types. Our contextual rewriting approach improves grounding performance by 11-22 percentage points on average, with a pure detector (GroundingDINO) reaching 56.7% on pronominals after rewriting, nearly double the best end-to-end baseline. Results demonstrate that decoupling linguistic reasoning from visual perception is more effective than end-to-end approaches for conversational grounding.

Abstract

Visual Grounding

Explore similar work

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction