Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries without relying on task-specific training data, demanding strong visual understanding capabilities. Existing Vision-Language Models~(VLMs), such as CLIP, commonly address zero-shot REC by directly measuring feature similarities between textual queries and image regions. However, these methods struggle to capture fine-grained visual details and understand complex object relationships. Meanwhile, Large Language Models~(LLMs) excel at high-level semantic reasoning, their inability to directly abstract visual features into textual semantics limits their application in REC tasks. To overcome these limitations, we propose SGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries. Specifically, we first employ a VLM to construct a query-driven scene graph that explicitly encodes spatial relationships, descriptive captions, and object interactions relevant to the given query. By leveraging this scene graph, we bridge the gap between low-level image regions and higher-level semantic understanding required by LLMs. Finally, an LLM infers the target object from the structured textual representation provided by the scene graph, responding with detailed explanations for its decisions that ensure interpretability in the inference process. Extensive experiments show that SGREC achieves top-1 accuracy on most zero-shot REC benchmarks, including RefCOCO val (66.78%), RefCOCO+ testB (53.43%), and RefCOCOg val (73.28%), highlighting its strong visual scene understanding.
Figures & tables
Fig. 1: Problem definition of zero-shot referring expression comprehension.
Fig. 2: Considering the spatial information, object appearance, and semantic relationships within the query, the generated query-driven scene graphs employ the relevant objects’ coordinates, captions, and visual interactions to describe visual scenes, facilitating inference by LLMs comprehensively. The queries from the datasets are natural and written in an informal style.
Fig. 3: Pipeline of the proposed SGREC. In Step 1 , SGREC begins by extracting nouns, predicting categories, and inferring subjects from the input query and the original image. It then identifies query-related objects by selecting those with matched labels. In Step 2 , a scene graph is generated in three parts: class names and coordinates from the detector to encode spatial information, generated image captions for each object to describe their appearance, and predicted relation triplets to capture interactions between objects. Finally, in Step 3 , SGREC analyzes the query and the generated scene graph to infer the index of the target object.
Fig. 4: Detailed prompts used in SGREC, covering modules for subject inference, object caption generation, interaction extraction, and final LLM-based inference.
Fig. 5: Illustration of the subject inference, including its inputs and outputs.
Fig. 6: We highlight the obj1 and the obj2 using different colors to reduce confusion in scenarios where multiple similar objects appear.
Method
Inference Model
RefCOCO
RefCOCO+
RefCOCOg
Avg
val
testA
testB
val
testA
testB
val
test
GPT-4V [ 43 ]
GPT-4V
25.48
26.22
24.39
10.59
18.23
8.87
14.26
15.42
17.93
CPT [ 44 ]
VinVL
32.20
36.10
30.30
31.90
35.20
28.80
36.70
36.50
33.46
ReCLIP [ 7 ]
CLIP ViT-B
45.78
46.10
47.07
47.87
50.10
45.10
59.33
59.01
50.05
RedCircle [ 8 ]
CLIP ViT-L
49.80
58.60
39.90
55.30
63.90
45.40
59.40
58.90
53.90
Pseudo-Q [ 42 ]
TransVG
56.02
58.25
54.13
38.88
45.06
32.13
46.25
47.44
47.27
TABLE I: Performance comparison of top-1 accuracy (%) with zero-shot REC methods on RefCOCO, RefCOCO+, and RefCOCOg datasets. The best and second-best results are boldfaced and underlined , respectively. “Avg” denotes the average performance across all splits.
Model
Setting
RefCOCO
RefCOCO+
RefCOCOg
Avg
val
testA
testB
val
testA
testB
val
test
LGRAN [ 33 ]
Fully-supervised
82.00
81.20
84.00
66.60
67.60
65.50
75.40
74.70
74.62
SGMN [ 32 ]
-
86.67
85.36
-
78.66
69.77
-
81.42
80.37
Ferret-13B [ 27 ]
89.48
92.41
84.36
82.81
88.14
75.17
85.83
86.34
85.57
Grounding-DINO [ 46 ]
90.56
93.19
88.24
82.75
88.95
75.92
86.13
87.02
86.60
Cycle-Free [ 47 ]
Weakly-supervised
39.58
41.46
37.96
39.19
39.63
37.53
-
-
39.23
TABLE II: Performance comparison under zero-shot, weakly supervised, and fully supervised settings on RefCOCO/+/g datasets. The best results are boldfaced . “Avg” denotes the average performance across all splits.
Method
Inference
RefCOCO
RefCOCO+
RefCOCOg
Avg
val
testA
testB
val
testA
testB
val
test
SGREC
LLaMA-8B [ 50 ]
53.41
57.58
48.95
48.74
52.69
44.43
59.72
59.13
53.08
LLaMA-70B [ 50 ]
64.04
69.26
59.82
55.85
60.93
51.16
71.28
71.23
62.95
LLaVA-7B [ 37 ]
45.58
50.72
42.32
45.37
48.67
42.58
56.64
56.38
48.53
LLaVA-72B [ 37 ]
65.86
70.28
61.76
57.53
61.35
53.77
73.10
72.86
64.56
Qwen-7B [ 51 ]
54.15
58.26
49.40
49.54
53.53
46.10
64.06
64.11
54.89
TABLE III: Influence of different inference models on RefCOCO, RefCOCO+, and RefCOCOg datasets. The best and second-best results are boldfaced and underlined , respectively. “Avg” denotes the average performance across all eight splits.
Detected objects
RefCOCO
RefCOCO+
RefCOCOg
Noun
Cat
Sub
✓
58.11
49.00
68.26
✓
✓
62.68
53.28
71.67
✓
✓
✓
66.78
57.17
73.28
TABLE IV: Ablation Study of Detected Objects on val split on RefCOCO/+/g. Cat : Predicted Category. Sub : Inferred Subject.
Scene Graph
RefCOCO
RefCOCO+
RefCOCOg
Det
Cap
Inter
✓
62.27
47.68
63.95
✓
✓
65.82
54.60
70.30
✓
✓
✓
66.78
57.17
73.28
TABLE V: Ablation Study of different information contained in generated scene graphs on val split of RefCOCO/+/g. Det : Detected objects along with their coordinates and attribute information. Cap : object captions. Inter : semantic interactions.
τ
RefCOCO
RefCOCO+
RefCOCOg
0.3
66.64
56.81
73.03
0.4
66.73
57.01
73.14
0.5
66.78
57.17
73.28
0.6
66.37
55.98
72.06
0.7
65.19
55.43
71.52
TABLE VI: Influence of τ that retain detected objects on the val split of RefCOCO/+/g.
Fig. 7: The frequency of noun occurrences in queries across different datasets. (‘name’, number) indicates how many times the corresponding noun appears.
θ
RefCOCO
RefCOCO+
RefCOCOg
0.1
66.62
56.39
72.94
0.2
66.78
57.17
73.28
0.3
66.59
56.38
72.32
0.4
66.46
56.16
72.27
0.5
66.25
55.84
71.86
TABLE VII: Influence of θ that identify possible interactions on the val split of RefCOCO/+/g.
Freq of Nouns
RefCOCO
RefCOCO+
RefCOCOg
Queries
Acc
Queries
Acc
Queries
Acc
>200
5255
70.50
4223
61.61
2043
77.43
100 ∼ 200
2551
66.13
2560
57.34
1163
72.92
<=100
3028
60.87
3975
52.33
1690
68.52
TABLE VIII: Model performance comparisons between high-frequency and low-frequency nouns on the val split of RefCOCO/+/g. Acc denotes accuracy.
Temperature
Top_p
RefCOCOg val
Default
0.7
0.8
73.28
Exp1
0.7
1.0
72.92
Exp2
0.3
0.8
73.02
Exp2
0.3
1.0
73.24
TABLE IX: Accuracy across different decoding parameter settings on the val split of RefCOCOg.
Fig. 8: Distribution of detected query-related objects across image-query pairs on the val split of RefCOCOg.
Num of Objects
RefCOCOg
Samples
Acc
[0,5)
2089
74.63
[5,10)
2045
73.49
[10,15)
527
70.78
[15,20)
162
59.87
≥ 20
59
77.97
TABLE X: Model performance comparison between sparse and dense scenes on the val split of RefCOCOg. Acc denotes accuracy. Samples denotes the number of image-query pairs.
Type
Example
RefCOCOg val
Natural language
‘a man wearing a red…’
62.01
Structured text
Object 1. label: man, attribute:…
72.79
JSON
{‘id’:1, ‘label’:[man], ‘attribute’:[…] }
73.28
TABLE XI: Examples of three kinds of input forms and corresponding results on the RefCOCOg val.
Fig. 9: Correct comprehension of the detected targets and LLM explanations on RefCOCO, RefCOCO+, and RefCOCOg. The red bounding boxes indicate the ground truth annotations, and the blue bounding boxes depict our predicted results.
Fig. 10: Failure case visualization. The red bounding boxes indicate the ground truth annotations, and the blue bounding boxes depict our predicted results.
Error rate
RefCOCO
RefCOCO+
RefCOCOg
Missed detection
10.9%
12.2%
8.6%
Labeling error rate [ 52 ]
14%
24%
5%
LLM misinterpretation
7.6%
6.4%
13.2%
TABLE XII: Analysis of error types within the framework. Reported rates (%) correspond to averages over the sub-splits of each dataset.
Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.
Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.
Lian Xu, Mohammed Bennamoun, Farid Boussaid +3
The University of Western Australia · Murdoch University · Sun Yat-sen University +1
Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios. OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions. Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC. Project page: https://zongjianwu.github.io/openref
Zongjian Wu, Lei Zhang
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China