Generalised referring expression comprehension (GREC) aims to identify and localise the zero, one or multiple objects referred to by a language expression. In robotic manipulation, grounding also needs to remain accurate when requests for the same objects are paraphrased or embedded in additional context in robotics. However, grounding expert models are sensitive to instruction variation, while generative vision--language models (VLMs) struggle with complete target recovery and rejection of absent targets. These complementary limitations motivate combining the contextual representations of VLMs with the direct spatial prediction capabilities of grounding experts. We therefore introduce VLM-guided Grounding Expert (VGE), which combines a frozen VLM with a trainable grounding expert through layerwise joint attention and text-conditioned instance scoring. These interfaces leverage multimodal understanding of the VLM to guide object-query refinement and candidate selection while retaining direct bounding-box prediction. Across three datasets, VGE achieves the highest averaged F1 score among the evaluated methods on all five test splits, with differences of at most 1.2 percentage points across instruction conditions within each split. On the dataset with robotic scenes, VGE also achieves the highest F1 across instruction variants and no-target, single-target and multiple-target cases. VGE provides a 5.3--13.0× inference speedup over the baseline with the same VLM base model. These results support combining VLM context with specialised spatial prediction to address both grounding challenges, with particularly strong performance in robotic scenes. Code will be released upon publication.
Figures & tables
Figure 1: Challenges in robotic grounding. (a) Instruction variation can change predictions despite an unchanged target. (b) Target-set errors include selecting absent objects and missing requested instances. Dashed teal boxes mark intended targets; solid orange boxes mark illustrative predictions. All examples are schematic and use the same scene.
Figure 2: VGE architecture. A frozen VLM supplies layerwise joint attention and contextualised text features to a trainable grounding expert. Gated joint attention refines object queries, whilst text-conditioned scoring adjusts native matching logits. The box prediction head produces candidate boxes; target selection combines boxes and scores into the predicted set.
Figure 3: Grounding without task-specific fine-tuning. Mean F1 (%) on the Test A splits of gRefCOCO and RoboRefIt. (a) Original, paraphrased and context-augmented instructions. (b) No-target, single-target and multiple-target examples under original instructions. Qwen denotes Qwen3.5. Detector thresholds are determined on the validation set for each dataset. Error bars show one bootstrapped standard deviation.
Table 1: Grounding across instruction variants and varying numbers of targets. Mean sample F1 (%) on three datasets. Subscripts report bootstrap standard deviations from 1,000 reference-level resamples. Bold and underlined values indicate the highest and second-highest displayed scores in each column, respectively, with ties receiving the same formatting.
Figure 4: Qualitative grounding comparisons. (a) Predictions under original and context-augmented instructions with unchanged targets. (b) Examples with no, one and multiple targets. Dashed blue boxes mark ground truth; solid teal and orange boxes mark matched and unmatched predictions, respectively. TP/FP/FN: true positives, false positives and false negatives. DINO denotes Grounding DINO; LoRA denotes the Qwen3.5-9B baseline with LoRA adaptation.
gRefCOCO
RoboRefIt
Setting
Test A
Test B
Test A
Test B
Frozen
79.98±0.41
72.52±0.55
95.02±0.15
89.67±0.26
Expert-gradient
80.55±0.40
73.64±0.54
95.17±0.15
89.27±0.27
CE-adapted
81.95±0.38
77.14±0.50
94.20±0.15
89.16±0.26
Adapted-frozen
81.95±0.38
75.78±0.50
94.97±0.15
89.96±0.25
Table 2: Effect of VLM adaptation on grounding. Mean sample F1 (%), averaged equally over original, paraphrased and context-augmented instructions.
Dataset
Method
Mean
P50
P95
gRefCOCO
Grounding DINO
68.1
67.0
72.3
Qwen3.5-9B(LoRA)
1820.8
728.4
2561.5
VGE
139.9
135.4
174.3
RoboRefIt
Grounding DINO
66.4
65.5
69.1
Qwen3.5-9B(LoRA)
737.5
736.0
1396.5
VGE
140.0
138.0
145.1
Table 3: Inference latency. Values are in milliseconds, measured on identical inputs and hardware, excluding image and text preprocessing. P50/P95: median/95th-percentile latency.
gRefCOCO
RoboRefIt
Setting
Test A
Test B
Test A
Test B
DINO-only
69.07±0.45
60.39±0.57
82.25±0.17
76.29±0.26
Joint-attn-only
76.70±0.39
71.49±0.49
79.66±0.16
73.47±0.25
Scoring-only
78.51±0.40
72.91±0.52
83.61±0.15
78.29±0.24
VGE
79.98±0.41
72.52±0.55
95.02±0.15
89.67±0.26
Table 4: Contributions of the two interfaces. Mean sample F1 (%), averaged equally over the three instruction conditions.
Dataset
Split
Images
Refs.
None
Single
Multiple
gRefCOCO
Train
17,709
81,966
12,564
42,004
27,398
Validation
785
1,200
400
400
400
Test A
750
6,265
749
1,975
3,541
Test B
750
4,618
750
1,807
2,061
RoboRefIt
Train
7,529
69,820
7,529
54,351
7,940
Validation
400
1,200
400
400
400
Table S1 : Dataset statistics. Unique images and references in our experimental splits. None, Single and Multiple count references with zero, one and multiple ground-truth targets, respectively.
Figure S1 : Effect of VLM and grounding-expert choices. Mean sample F1 (%), averaged equally over original, paraphrased and context-augmented instructions. (a) Frozen VLM variants with Grounding DINO-base fixed. (b) Grounding-expert variants with frozen Qwen3.5-9B fixed. Error bars show standard deviations from 1,000 paired reference-bootstrap resamples.
Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.
Aernaer Akelijiang, Jiannan Li, Zhineng Chen +2
Singapore Management University · Fudan University
Grounding language in the physical world requires AI systems to interpret references that emerge dynamically during conversation. While current vision-language models (VLMs) excel at static image tasks, they struggle to resolve ambiguous expressions in spontaneous, multi-turn dialogue. We address this gap by introducing (1) a benchmark for referential communication in dynamic 3D environments, built from 6.7 hours of egocentric VR interaction with synchronized speech, motion, gaze, and 3D scene geometry, and (2) a two-stage grounding pipeline that explicitly resolves conversational ambiguity before visual localization. The benchmark includes over 4,200 manually verified referring expressions spanning full, partitive, and pronominal types. Our contextual rewriting approach improves grounding performance by 11-22 percentage points on average, with a pure detector (GroundingDINO) reaching 56.7% on pronominals after rewriting, nearly double the best end-to-end baseline. Results demonstrate that decoupling linguistic reasoning from visual perception is more effective than end-to-end approaches for conversational grounding.
Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan +4
KTH Royal Institute of Technology, Stockholm, Sweden
While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.
Yanyan Zhang, Disheng Liu, Xinpeng Li +8
Case Western Reserve University Cleveland, OH, USA