Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs

Yike Wu; Necva Bolucu; Stephen Wan; Dadong Wang; Jiahao Xia; Jian Zhang

cs.CVMar 26, 2026

Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs

Authors: Yike Wu, Necva Bolucu, Stephen Wan, Dadong Wang, Jiahao Xia, Jian Zhang

Abstract

Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries without relying on task-specific training data, demanding strong visual understanding capabilities. Existing Vision-Language Models~(VLMs), such as CLIP, commonly address zero-shot REC by directly measuring feature similarities between textual queries and image regions. However, these methods struggle to capture fine-grained visual details and understand complex object relationships. Meanwhile, Large Language Models~(LLMs) excel at high-level semantic reasoning, their inability to directly abstract visual features into textual semantics limits their application in REC tasks. To overcome these limitations, we propose SGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries. Specifically, we first employ a VLM to construct a query-driven scene graph that explicitly encodes spatial relationships, descriptive captions, and object interactions relevant to the given query. By leveraging this scene graph, we bridge the gap between low-level image regions and higher-level semantic understanding required by LLMs. Finally, an LLM infers the target object from the structured textual representation provided by the scene graph, responding with detailed explanations for its decisions that ensure interpretability in the inference process. Extensive experiments show that SGREC achieves top-1 accuracy on most zero-shot REC benchmarks, including RefCOCO val (66.78%), RefCOCO+ testB (53.43%), and RefCOCOg val (73.28%), highlighting its strong visual scene understanding.

Figures & tables

Fig. 1: Problem definition of zero-shot referring expression comprehension.

Fig. 2: Considering the spatial information, object appearance, and semantic relationships within the query, the generated query-driven scene graphs employ the relevant objects’ coordinates, captions, and visual interactions to describe visual scenes, facilitating inference by LLMs comprehensively. The queries from the datasets are natural and written in an informal style.

Fig. 3: Pipeline of the proposed SGREC. In Step 1 , SGREC begins by extracting nouns, predicting categories, and inferring subjects from the input query and the original image. It then identifies query-related objects by selecting those with matched labels. In Step 2 , a scene graph is generated in three parts: class names and coordinates from the detector to encode spatial information, generated image captions for each object to describe their appearance, and predicted relation triplets to capture interactions between objects. Finally, in Step 3 , SGREC analyzes the query and the generated scene graph to infer the index of the target object.

Fig. 4: Detailed prompts used in SGREC, covering modules for subject inference, object caption generation, interaction extraction, and final LLM-based inference.

Fig. 5: Illustration of the subject inference, including its inputs and outputs.

Fig. 6: We highlight the obj1 and the obj2 using different colors to reduce confusion in scenarios where multiple similar objects appear.

Method	Inference Model	RefCOCO			RefCOCO+			RefCOCOg		Avg
Method	Inference Model	val	testA	testB	val	testA	testB	val	test	Avg
GPT-4V [ 43 ]	GPT-4V	25.48	26.22	24.39	10.59	18.23	8.87	14.26	15.42	17.93
CPT [ 44 ]	VinVL	32.20	36.10	30.30	31.90	35.20	28.80	36.70	36.50	33.46
ReCLIP [ 7 ]	CLIP ViT-B	45.78	46.10	47.07	47.87	50.10	45.10	59.33	59.01	50.05
RedCircle [ 8 ]	CLIP ViT-L	49.80	58.60	39.90	55.30	63.90	45.40	59.40	58.90	53.90
Pseudo-Q [ 42 ]	TransVG	56.02	58.25	54.13	38.88	45.06	32.13	46.25	47.44	47.27

TABLE I: Performance comparison of top-1 accuracy (%) with zero-shot REC methods on RefCOCO, RefCOCO+, and RefCOCOg datasets. The best and second-best results are boldfaced and underlined , respectively. “Avg” denotes the average performance across all splits.

Model	Setting	RefCOCO			RefCOCO+			RefCOCOg		Avg
Model	Setting	val	testA	testB	val	testA	testB	val	test	Avg
LGRAN [ 33 ]	Fully-supervised	82.00	81.20	84.00	66.60	67.60	65.50	75.40	74.70	74.62
SGMN [ 32 ]		-	86.67	85.36	-	78.66	69.77	-	81.42	80.37
Ferret-13B [ 27 ]		89.48	92.41	84.36	82.81	88.14	75.17	85.83	86.34	85.57
Grounding-DINO [ 46 ]		90.56	93.19	88.24	82.75	88.95	75.92	86.13	87.02	86.60
Cycle-Free [ 47 ]	Weakly-supervised	39.58	41.46	37.96	39.19	39.63	37.53	-	-	39.23

TABLE II: Performance comparison under zero-shot, weakly supervised, and fully supervised settings on RefCOCO/+/g datasets. The best results are boldfaced . “Avg” denotes the average performance across all splits.

Method	Inference	RefCOCO			RefCOCO+			RefCOCOg		Avg
Method	Inference	val	testA	testB	val	testA	testB	val	test	Avg
SGREC	LLaMA-8B [ 50 ]	53.41	57.58	48.95	48.74	52.69	44.43	59.72	59.13	53.08
	LLaMA-70B [ 50 ]	64.04	69.26	59.82	55.85	60.93	51.16	71.28	71.23	62.95
	LLaVA-7B [ 37 ]	45.58	50.72	42.32	45.37	48.67	42.58	56.64	56.38	48.53
	LLaVA-72B [ 37 ]	65.86	70.28	61.76	57.53	61.35	53.77	73.10	72.86	64.56
	Qwen-7B [ 51 ]	54.15	58.26	49.40	49.54	53.53	46.10	64.06	64.11	54.89

TABLE III: Influence of different inference models on RefCOCO, RefCOCO+, and RefCOCOg datasets. The best and second-best results are boldfaced and underlined , respectively. “Avg” denotes the average performance across all eight splits.

Detected objects			RefCOCO	RefCOCO+	RefCOCOg
Noun	Cat	Sub	RefCOCO	RefCOCO+	RefCOCOg
$\checkmark$			58.11	49.00	68.26
$\checkmark$	$\checkmark$		62.68	53.28	71.67
$\checkmark$	$\checkmark$	$\checkmark$	66.78	57.17	73.28

TABLE IV: Ablation Study of Detected Objects on val split on RefCOCO/+/g. Cat : Predicted Category. Sub : Inferred Subject.

Scene Graph			RefCOCO	RefCOCO+	RefCOCOg
Det	Cap	Inter	RefCOCO	RefCOCO+	RefCOCOg
$\checkmark$			62.27	47.68	63.95
$\checkmark$	$\checkmark$		65.82	54.60	70.30
$\checkmark$	$\checkmark$	$\checkmark$	66.78	57.17	73.28

TABLE V: Ablation Study of different information contained in generated scene graphs on val split of RefCOCO/+/g. Det : Detected objects along with their coordinates and attribute information. Cap : object captions. Inter : semantic interactions.

$\tau$	RefCOCO	RefCOCO+	RefCOCOg
0.3	66.64	56.81	73.03
0.4	66.73	57.01	73.14
0.5	66.78	57.17	73.28
0.6	66.37	55.98	72.06
0.7	65.19	55.43	71.52

TABLE VI: Influence of

\tau

that retain detected objects on the val split of RefCOCO/+/g.

Fig. 7: The frequency of noun occurrences in queries across different datasets. (‘name’, number) indicates how many times the corresponding noun appears.

$\theta$	RefCOCO	RefCOCO+	RefCOCOg
0.1	66.62	56.39	72.94
0.2	66.78	57.17	73.28
0.3	66.59	56.38	72.32
0.4	66.46	56.16	72.27
0.5	66.25	55.84	71.86

TABLE VII: Influence of

\theta

that identify possible interactions on the val split of RefCOCO/+/g.

Freq of Nouns	RefCOCO		RefCOCO+		RefCOCOg
Freq of Nouns	Queries	Acc	Queries	Acc	Queries	Acc
>200	5255	70.50	4223	61.61	2043	77.43
100 $\sim$ 200	2551	66.13	2560	57.34	1163	72.92
<=100	3028	60.87	3975	52.33	1690	68.52

TABLE VIII: Model performance comparisons between high-frequency and low-frequency nouns on the val split of RefCOCO/+/g. Acc denotes accuracy.

	Temperature	Top_p	RefCOCOg val
Default	0.7	0.8	73.28
Exp1	0.7	1.0	72.92
Exp2	0.3	0.8	73.02
Exp2	0.3	1.0	73.24

TABLE IX: Accuracy across different decoding parameter settings on the val split of RefCOCOg.

Fig. 8: Distribution of detected query-related objects across image-query pairs on the val split of RefCOCOg.

Num of Objects	RefCOCOg
Num of Objects	Samples	Acc
$[0,5)$	2089	74.63
$[5,10)$	2045	73.49
$[10,15)$	527	70.78
$[15,20)$	162	59.87
$\geq$ 20	59	77.97

TABLE X: Model performance comparison between sparse and dense scenes on the val split of RefCOCOg. Acc denotes accuracy. Samples denotes the number of image-query pairs.

Type	Example	RefCOCOg val
Natural language	‘a man wearing a red…’	62.01
Structured text	Object 1. label: man, attribute:…	72.79
JSON	{‘id’:1, ‘label’:[man], ‘attribute’:[…] }	73.28

TABLE XI: Examples of three kinds of input forms and corresponding results on the RefCOCOg val.

Fig. 9: Correct comprehension of the detected targets and LLM explanations on RefCOCO, RefCOCO+, and RefCOCOg. The red bounding boxes indicate the ground truth annotations, and the blue bounding boxes depict our predicted results.

Fig. 10: Failure case visualization. The red bounding boxes indicate the ground truth annotations, and the blue bounding boxes depict our predicted results.

Error rate	RefCOCO	RefCOCO+	RefCOCOg
Missed detection	10.9%	12.2%	8.6%
Labeling error rate [ 52 ]	14%	24%	5%
LLM misinterpretation	7.6%	6.4%	13.2%

TABLE XII: Analysis of error types within the framework. Reported rates (%) correspond to averages over the sub-splits of each dataset.

Referring Expression Comprehension

See all 21 papers

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
Feb 27, 2026Qihua Dong, Kuo Yang, Lin Ju +6Referring Expression Comprehension VLM Evaluation
Northeastern University
Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension
Jul 6, 2026Lian Xu, Mohammed Bennamoun, Farid Boussaid +3Visual Representation Learning Vision-Language Grounding
The University of Western Australia · Murdoch University · Sun Yat-sen University +1
Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
May 25, 2026Zongjian Wu, Lei ZhangVLM Evaluation Vision-Language Grounding
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China

Feb 27, 2026J/K move · Enter open · S save

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

Qihua Dong, Kuo Yang, Lin Ju, Handong Zhao+5

Northeastern University

Referring Expression Comprehension VLM Evaluation Vision-Language Grounding Visually Grounded Reasoning

Open paper AskPDF arXiv

Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.

Abstract

Referring Expression Comprehension

Explore similar work

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension

Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker