Seeing Isn't Resolving: Visual and Linguistic Disambiguation with Conversational Grounding
Organizations: Signals and Interactive Systems Lab, University of Trento, Italy · The University of Osaka, Japan · Guardian Robot Project, RIKEN, Japan
Abstract
Visual grounding benchmarks usually do not consider the contextual uncertainty of real-world scenarios. In this study, we construct a novel benchmark, the Japanese evaluation for Ambiguous visual Grounding with Dialogue (JAG-D) dataset. It consists of human-robot conversational multimodal interactions covering different types of ambiguity, arising from visual and linguistic categories. We evaluate the capabilities of different state-of-the-art Visual Language Models (VLMs) for ambiguity recognition and resolution. We compare dialogue context grounding performance for visual and linguistic ambiguity resolution. Our experiments show that VLMs can substantially improve the resolution of linguistic ambiguity, especially Japanese zero anaphora, by using dialogue context. We further conduct a comparative analysis to understand whether grounding improvements arise from the semantic content of the dialogue or from its conversational structure.
Figures & tables
| Category | Count | % |
|---|---|---|
| Instruction Ambiguity | ||
| Ambiguous | 232 | 48.3 |
| same_object | 58 | 12.1 |
| same_attribute | 55 | 11.5 |
| pronoun | 61 | 12.7 |
| zero_anaphora | 58 | 12.1 |
| Visual Ambiguity (R) | Linguistic Ambiguity (R) | |||||
|---|---|---|---|---|---|---|
| Model | Overall (F1-M) | Non-Ambig (R) | Same Obj | Same Attr | Pronoun | Zero Ana |
| Qwen3-VL | 83.0 | 94.4 | 32.8 | 54.5 | 96.7 | 100.0 |
| Gemma3 | 78.5 | 85.9 | 39.7 | 52.7 | 96.7 | 93.1 |
| LLM-jp VILA | 60.1 | 67.3 | 46.6 | 54.5 | 47.5 | 63.8 |
| Sarashina2 | 40.3 | 7.7 | 98.3 | 98.2 | 100.0 | 100.0 |
| EvoVLM-JP | 32.6 | 0.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| Model | Phase | [email protected] (%) | [email protected] (%) | mIoU | #Correct |
|---|---|---|---|---|---|
| GLIP | Instruction-Only | 26.9 | 24.4 | 0.272 | 129 |
| GLIP | Dialogue-Enhanced | 53.3 | 49.4 | 0.501 | 256 |
| MDETR | Instruction-Only | 25.4 | 17.9 | 0.239 | 122 |
| MDETR | Dialogue-Enhanced | 54.0 | 32.9 | 0.449 | 259 |
| Qwen3-VL | Instruction-Only | 63.5 | 56.5 | 0.585 | 305 |
| Qwen3-VL | Dialogue-Enhanced | 79.0 | 68.3 | 0.717 | 379 |
| Ambiguity Type | GLIP | MDETR | Qwen3-VL |
|---|---|---|---|
| Non-ambiguous | 61.7\,({\color[rgb]{0,0.88,0}\uparrow+21.8}) | 63.3\,({\color[rgb]{0,0.88,0}\uparrow+24.2}) | 87.9\,({\color[rgb]{0,0.88,0}\uparrow+0.4}) |
| Ambiguous | 44.4\,({\color[rgb]{0,0.88,0}\uparrow+31.5}) | 44.0\,({\color[rgb]{0,0.88,0}\uparrow+33.2}) | 69.4\,({\color[rgb]{0,0.88,0}\uparrow+31.5}) |
| same_object | 32.8\,({\color[rgb]{0,0.88,0}\uparrow+12.1}) | 25.9\,({\color[rgb]{0,0.88,0}\uparrow+12.1}) | 62.1\,({\color[rgb]{0,0.88,0}\uparrow+6.9}) |
| same_attribute | 38.2\,({\color[rgb]{0,0.88,0}\uparrow+18.2}) | 38.2\,({\color[rgb]{0,0.88,0}\uparrow+14.5}) | 67.3\,({\color[rgb]{0,0.88,0}\uparrow+14.5}) |
| pronoun | 52.5\,({\color[rgb]{0,0.88,0}\uparrow+44.3}) | 54.1\,({\color[rgb]{0,0.88,0}\uparrow+50.8}) | 63.9\,({\color[rgb]{0,0.88,0}\uparrow+42.6}) |
| zero_anaphora | 53.4\,({\color[rgb]{0,0.88,0}\uparrow+50.0}) | 56.9\,({\color[rgb]{0,0.88,0}\uparrow+53.4}) | 84.5\,({\color[rgb]{0,0.88,0}\uparrow+60.3}) |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Dialogue Mode | Final Order | Dialogue History |
|---|---|---|
| resolving | “Hand me the rectangular item.” | “robot”: “I see a green backpack on the couch. Is there something you need from it?” “master”: “Yes, there’s an item inside the main compartment.” “robot”: “I notice a black electronic device between your hands as you adjust the zipper. Is that the item?” “master”: “Exactly, that’s the one I need.” “robot”: “It seems to be rectangular and could be a tablet or a laptop . Should I retrieve it for you?” “master”: “Yes, please hand me the rectangular item .” |
| non-resolving | “Hand me the mug.” | “robot”: “I noticed there are two white mugs on the coffee table.” “master”: “Yes, they both look quite similar , don’t they?” “robot”: “Indeed. One is sitting right next to the yellow tissue box.” “master”: “And the other is right beside it, in front of me.” “robot”: “They’re both conveniently placed on the black coffee table.” “master”: “Could you hand me the mug ?” |
| supportive | “Hand me the notebook.” | “robot”: “I’m here to assist. Is there something specific you need today?” “master”: “Yes, I need the notebook . It’s on the couch.” “robot”: “I see multiple items. Is it near anything in particular?” “master”: “The notebook is dark blue and should be to the left of the green backpack .” “robot”: “Ah, and is it under a piece of white paper with Japanese text ?” “master”: “Yes, that’s the one. Hand me the notebook .” |
| Model | Prompt | Lang | Acc (%) | Prec (%) | Rec (%) | F1 (%) | F1-M (%) |
|---|---|---|---|---|---|---|---|
| Qwen3-VL | Few-shot | EN | 83.3 | 92.2 | 71.6 | 80.6 | 83.0 |
| Qwen3-VL | Few-shot | JA | 82.5 | 92.5 | 69.4 | 79.3 | 82.1 |
| Gemma3 | Few-shot | EN | 78.8 | 82.5 | 71.1 | 76.4 | 78.5 |
| Qwen3-VL | Zero-shot | JA | 76.9 | 79.5 | 70.3 | 74.6 | 76.7 |
| Gemma3 | Few-shot | JA | 76.5 | 70.4 | 88.4 | 78.4 | 76.3 |
| Gemma3 | Zero-shot | JA | 74.2 | 77.8 | 65.1 | 70.9 | 73.8 |
| Non-Ambig | Visual Ambiguity | Linguistic Ambiguity | ||||
| Model | Config | Recall | Same Obj | Same Attr | Pronoun | Zero Ana |
| Qwen3-VL | FS-EN | 94.4 | 32.8 | 54.5 | 96.7 | 100.0 |
| Qwen3-VL | FS-JA | 94.8 | 32.8 | 52.7 | 95.1 | 94.8 |
| Qwen3-VL | ZS-JA | 83.1 | 56.9 | 41.8 | 86.9 | 93.1 |
| Qwen3-VL | ZS-EN | 96.8 | 6.9 | 14.5 | 72.1 | 67.2 |
| Gemma3 | FS-EN | 85.9 | 39.7 | 52.7 | 96.7 | 93.1 |
| Intervention | Ambiguous | Non-ambiguous | ||||
|---|---|---|---|---|---|---|
| mIoU | IoU | Acc@ | mIoU | IoU | Acc@ | |
| Full dialogue (baseline) | 0.641 | 0.000 | 0.694 | 0.787 | 0.000 | 0.879 |
| Neutral substitution | 0.349 | -0.292 | 0.362 | 0.791 | 0.004 | 0.887 |
| Order only | 0.361 | -0.280 | 0.379 | 0.794 | 0.007 | 0.875 |
| Irrelevant substitution | 0.362 | -0.279 | 0.379 | 0.786 | -0.001 | 0.883 |
| First turns | 0.467 | -0.174 | 0.487 | 0.786 | -0.001 | 0.879 |
Vision-Language Grounding
Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.
See all 444 papers