Generalised referring expression comprehension (GREC) aims to identify and localise the zero, one or multiple objects referred to by a language expression. In robotic manipulation, grounding also needs to remain accurate when requests for the same objects are paraphrased or embedded in additional context in robotics. However, grounding expert models are sensitive to instruction variation, while generative vision--language models (VLMs) struggle with complete target recovery and rejection of absent targets. These complementary limitations motivate combining the contextual representations of VLMs with the direct spatial prediction capabilities of grounding experts. We therefore introduce VLM-guided Grounding Expert (VGE), which combines a frozen VLM with a trainable grounding expert through layerwise joint attention and text-conditioned instance scoring. These interfaces leverage multimodal understanding of the VLM to guide object-query refinement and candidate selection while retaining direct bounding-box prediction. Across three datasets, VGE achieves the highest averaged F1 score among the evaluated methods on all five test splits, with differences of at most 1.2 percentage points across instruction conditions within each split. On the dataset with robotic scenes, VGE also achieves the highest F1 across instruction variants and no-target, single-target and multiple-target cases. VGE provides a 5.3--13.0× inference speedup over the baseline with the same VLM base model. These results support combining VLM context with specialised spatial prediction to address both grounding challenges, with particularly strong performance in robotic scenes. Code will be released upon publication.
Figures & tables
Figure 1: Challenges in robotic grounding. (a) Instruction variation can change predictions despite an unchanged target. (b) Target-set errors include selecting absent objects and missing requested instances. Dashed teal boxes mark intended targets; solid orange boxes mark illustrative predictions. All examples are schematic and use the same scene.
Figure 2: VGE architecture. A frozen VLM supplies layerwise joint attention and contextualised text features to a trainable grounding expert. Gated joint attention refines object queries, whilst text-conditioned scoring adjusts native matching logits. The box prediction head produces candidate boxes; target selection combines boxes and scores into the predicted set.
Figure 3: Grounding without task-specific fine-tuning. Mean F1 (%) on the Test A splits of gRefCOCO and RoboRefIt. (a) Original, paraphrased and context-augmented instructions. (b) No-target, single-target and multiple-target examples under original instructions. Qwen denotes Qwen3.5. Detector thresholds are determined on the validation set for each dataset. Error bars show one bootstrapped standard deviation.
Table 1: Grounding across instruction variants and varying numbers of targets. Mean sample F1 (%) on three datasets. Subscripts report bootstrap standard deviations from 1,000 reference-level resamples. Bold and underlined values indicate the highest and second-highest displayed scores in each column, respectively, with ties receiving the same formatting.
Figure 4: Qualitative grounding comparisons. (a) Predictions under original and context-augmented instructions with unchanged targets. (b) Examples with no, one and multiple targets. Dashed blue boxes mark ground truth; solid teal and orange boxes mark matched and unmatched predictions, respectively. TP/FP/FN: true positives, false positives and false negatives. DINO denotes Grounding DINO; LoRA denotes the Qwen3.5-9B baseline with LoRA adaptation.
gRefCOCO
RoboRefIt
Setting
Test A
Test B
Test A
Test B
Frozen
79.98±0.41
72.52±0.55
95.02±0.15
89.67±0.26
Expert-gradient
80.55±0.40
73.64±0.54
95.17±0.15
89.27±0.27
CE-adapted
81.95±0.38
77.14±0.50
94.20±0.15
89.16±0.26
Adapted-frozen
81.95±0.38
75.78±0.50
94.97±0.15
89.96±0.25
Table 2: Effect of VLM adaptation on grounding. Mean sample F1 (%), averaged equally over original, paraphrased and context-augmented instructions.
Dataset
Method
Mean
P50
P95
gRefCOCO
Grounding DINO
68.1
67.0
72.3
Qwen3.5-9B(LoRA)
1820.8
728.4
2561.5
VGE
139.9
135.4
174.3
RoboRefIt
Grounding DINO
66.4
65.5
69.1
Qwen3.5-9B(LoRA)
737.5
736.0
1396.5
VGE
140.0
138.0
145.1
Table 3: Inference latency. Values are in milliseconds, measured on identical inputs and hardware, excluding image and text preprocessing. P50/P95: median/95th-percentile latency.
gRefCOCO
RoboRefIt
Setting
Test A
Test B
Test A
Test B
DINO-only
69.07±0.45
60.39±0.57
82.25±0.17
76.29±0.26
Joint-attn-only
76.70±0.39
71.49±0.49
79.66±0.16
73.47±0.25
Scoring-only
78.51±0.40
72.91±0.52
83.61±0.15
78.29±0.24
VGE
79.98±0.41
72.52±0.55
95.02±0.15
89.67±0.26
Table 4: Contributions of the two interfaces. Mean sample F1 (%), averaged equally over the three instruction conditions.
Dataset
Split
Images
Refs.
None
Single
Multiple
gRefCOCO
Train
17,709
81,966
12,564
42,004
27,398
Validation
785
1,200
400
400
400
Test A
750
6,265
749
1,975
3,541
Test B
750
4,618
750
1,807
2,061
RoboRefIt
Train
7,529
69,820
7,529
54,351
7,940
Validation
400
1,200
400
400
400
Table S1 : Dataset statistics. Unique images and references in our experimental splits. None, Single and Multiple count references with zero, one and multiple ground-truth targets, respectively.
Figure S1 : Effect of VLM and grounding-expert choices. Mean sample F1 (%), averaged equally over original, paraphrased and context-augmented instructions. (a) Frozen VLM variants with Grounding DINO-base fixed. (b) Grounding-expert variants with frozen Qwen3.5-9B fixed. Error bars show standard deviations from 1,000 paired reference-bootstrap resamples.