Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.
Figures & tables
Figure 1: Referring Scenario Comprehension (RSC) vs. traditional referring expression comprehension (REC). Each row shows the same target object under both paradigms. Traditional REC queries often name the target category directly, allowing success via lexical matching. RSC instead pairs each image with a lengthy scenario-based query specifying a user role, goal, and multiple disambiguating cues, including explicit contrasts against competing objects, and requires output identifying both the target object and its bounding box. The RSC difficulty tags (U/C/S/O/P: Uniqueness, Clutter, Size, Overlap, Position) characterize each instance, enabling fine-grained training and evaluation.
Benchmark
Train
ID Test
OOD Test
Referring Style
Query Style
Avg. ∣q∣
Reasoning Traces
Difficulty Tags
Competing Mentions
RefCOCO Kazemzadeh et al. (2014)
120,624
10,752
✗
Literal
Phrase
3.5
✗
✗
✗
RefCOCO+ Kazemzadeh et al. (2014)
120,191
10,615
✗
Literal
Phrase
3.5
✗
✗
✗
RefCOCOg Mao et al. (2016)
80,512
9,602
✗
Descriptive literal
Phrase
8.2
✗
✗
✓
Cops-Ref Chen et al. (2020)
119,603
12,586
✗
Compositional literal
Template
14.4
✗
✗
✓
Ref-Reasoning Yang et al. (2020)
721,164
34,609
✗
Literal
Template
8.5
Graph
Graph layout
✓
SK-VG Chen et al. (2023b)
23,403
6,597
✗
Scene knowledge Q&A
Q&A
5.3
✗
E/M/H
✓
Table 1: Comparison of referring expression grounding benchmarks. We compare existing datasets with RSC across dataset scale, evaluation splits, referring/query style, reasoning supervision, difficulty annotations, and competing-object mentions. Most prior datasets rely on short literal phrases and lack explicit reasoning traces or OOD evaluation. RSC introduces scenario-based queries written as natural language paragraphs and provides per-instance reasoning trace annotations. It further labels five interpretable difficulty factors (U/C/S/O/P) and includes both in-distribution (ID) and out-of-distribution (OOD) category test sets to evaluate reasoning and generalization.
Figure 2: Phase 1 filters and balances source instances computing five interpretable difficulty tags to form a tag-balanced candidate pool. Phase 2 generates annotations via a two-stage process: a small-scale refinement loop first validates the generation prompt through iterative system refinement and human audit, before large-scale scenario generation is applied to the full candidate pool. Each instance is annotated with a target object description, a scenario query with reasoning traces, acceptable category aliases, and an LLM-predicted bounding box for quality filtering. Phase 3 applies automatic and human quality control: a quality judge filters rough annotations, human auditors verify a stratified sample. The final RSC dataset provides, per instance, a scenario query, reasoning traces, acceptable names, ground-truth box, and difficulty tags.
Split
Initial
GT Filter
Kept
Quality Filter
Kept
Train (SFT)
30,000
29,551
98.5%
23,802
80.5%
Train (RL)
10,000
9,884
98.8%
7,540
76.3%
Test (ID)
5,000
4,915
98.3%
4,038
82.2%
Test (OOD)
5,000
3,983
79.7%
3,247
81.5%
Table 2: RSC data retention across quality control stages. After LLM GT Filter : instances passing the automatic schema, box IoU, alias consistency, and leakage gates. After Quality Filter : instances passing the full judge-and-audit pipeline.
Figure 3: ScenGround prompt and output schema. Given an image and a user-driven scenario, the model is instructed to reason inside <think> and to emit a structured JSON inside <answer> . The JSON contains target_object and bbox [x,y,w,h] . Scenarios avoid category names and force disambiguation via attributes, relations, and spatial cues, while <think> must justify the selection and ignore distractors. This schema is used in both TP-SFT and IC-GRPO. To increase the robustness of ScenGround, IC-GRPO uses 7 other templates.
Table 3: Performance on RSC (ID and OOD). Metrics: mIoU and Acc@{0.5,0.7} (higher is better); Cat Acc = category accuracy. ‡ Oracle settings, not directly comparable: Grounding DINO receives privileged inputs unavailable at inference: cat token provides the gold category name directly and ref. cue feeds the conciser reasoning trace conclusion "referring expression cue" (See Figure 2 ) as the open-vocabulary query.
Table 4: Ablation of curriculum in IC-GRPO on RSC in-domain (RSC-ID) and out-of-domain (RSC-OOD). Metrics: mIoU and Acc@{0.5,0.7} for boxes; Cat Acc for category naming. Single Stage trains on the union of all RL samples without curriculum; Stage- k trains on the stage- k slice, all other settings unchanged.
RefCOCO+
RefCOCOg
Method
Val
Test A
Test B
Val
Test
Qwen2.5 - VL
84.20
89.10
76.90
87.20
87.20
Qwen2.5 - VL ∗
52.54
56.75
52.53
52.46
51.40
ScenGround ∗ , SFT
50.86
43.84
50.74
65.45
63.39
ScenGround ∗ , GRPO
70.16
74.63
70.05
78.19
75.61
Table 5: Referring expression comprehension on RefCOCO+ and RefCOCOg. Metric: [email protected] (%, ↑ ). † Use the official grounding pipeline and prompt Bai et al. (2025a) . ∗ Use a custom prompt.
RSC-ID
RSC-OOD
Method
mIoU
Acc @0.5
Acc @0.7
mIoU
Acc @0.5
Acc @0.7
Qwen2.5-VL 7B
27.37
24.09
13.42
19.11
13.50
7.70
ScenGround, SFT
51.18
55.80
37.99
40.71
42.65
25.86
ScenGround, GRPO
50.90
54.38
37.01
40.11
40.30
24.50
Table 6: Localization under the box-only setting on RSC in-distribution (RSC-ID) and out-of-distribution (RSC-OOD). Metrics are mIoU and Acc@{0.5,0.7} .
Text Only
Text-Image
Standard
Method
ID
OOD
ID
OOD
ID
OOD
Qwen2.5-VL 7B
56.24
22.34
77.54
33.33
29.65
19.28
ScenGround-SFT
81.03
10.69
93.66
14.92
92.72
12.30
ScenGround-GRPO
85.13
20.22
94.73
22.68
93.71
20.67
Table 7: Category accuracy on RSC in-distribution (ID) and out-of-distribution (OOD) under three input–output settings. Text Only : predict category from scenario only. Text-Image : predict category from scenario+image (no box). Standard : full task.
Figure 4: Qualitative results on RSC. Red box for ground truth and blue box for ScenGround prediction.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Query length distribution. RSC queries are substantially longer than RefCOCO+ and RefCOCOg expressions, reflecting the paragraph-length scenario descriptions that specify user roles, goals, and disambiguating cues. RefCOCO+ and RefCOCOg peak below 10 words; RSC peaks around 50–60 words.
Figure 6: Instance size distribution. RSC covers a broader range of instance sizes than RefCOCO+ and RefCOCOg, with notably higher density at smaller instances.
Dataset
Unique Vocabulary
RefCOCO+
4,240
RefCOCOg
4,917
RSC (Ours)
9,086
Appendix
Table 8: Vocabulary size comparison. Number of unique words across all queries in each dataset’s test split.
Figure 7: Difficulty tag distributions for RSC-ID and RSC-OOD. Each panel shows the per-tag marginal distribution for the in-distribution (RSC-ID) and out-of-distribution (RSC-OOD) test splits across all five axes: Uniqueness (U), Clutter (C), Size (S), Overlap (O), and Position (P).
Figure 8: Top-10 words per part-of-speech category in RSC scenarios. Frequency counts of the most common nouns, verbs, adverbs, and prepositions across all RSC scenario queries.
Figure 9: Top 20 most frequent categories in RSC-ID and RSC-OOD test splits.
Figure 10: RSC-ID category frequency distribution. Frequency of all 79 COCO categories across 4,038 in-distribution test instances, plotted on a log scale. The distribution is moderately long-tailed, with common household objects dominating.
Figure 11: RSC-OOD category frequency distribution. Frequency of the top 80 most common categories among 395 LVIS categories across 3,247 out-of-distribution test instances, plotted on a log scale. The long-tailed distribution reflects LVIS’s diverse fine-grained vocabulary.
LLM-generated
Human-authored
Method
mIoU
Cat. Acc
mIoU
Cat. Acc
Qwen2.5-VL
33.4
31.0
31.3
32.0
ScenGround
58.0
96.0
59.2
98.0
Appendix
Table 9: Model performance on human-authored vs. LLM-generated scenarios for the same 100 RSC instances. The human-authored scenario query is randomly drawn from one of the three annotators. ScenGround performs comparably on human-authored queries, supporting ecological validity and confirming that training does not overfit to LLM-generated patterns.
Human-authored
LLM-generated
There is a tourist walking down the street who needs to cross at the intersection. You are looking for a green light on a pole, away from the main intersection light. It’s closer to the left side of the street and looks higher than the building.
As a pedestrian waiting to cross, you want to check if it’s safe. You look for the green-lit signal on the rightmost pole, above a blue sign, away from the main intersection lights. Its color, position, and separation from the cluster help you identify it.
A parent is watching from the beach, making sure the kid with dark hair is safe. You notice most of his body is submerged, he’s close to an adult, and he’s the leftmost in the group. Please identify the kid in the crowd.
As a parent watching from the shore, you want to check if your child, who has dark hair and is the leftmost in the group, is safe. They are closest to the water’s edge, mostly submerged, and positioned to the left of the shirtless swimmer.
A student is organizing the desk. They are looking for a green-handled item that sits on the desk. There are orange scissors and colorful pens in a mug, but those are not needed. Find the target object to the left of the mug that can be used to cut things.
As a teacher organizing supplies, you need the small green-handled item lying on the desk, not the larger orange-handled ones in the mug. Its bright green color, flat position, and separation from the mug make it easy to spot.
Appendix
Table 10: Human-authored vs. LLM-generated scenario pairs for the same targets. Both convey equivalent referential content through different surface forms.
Figure 12: The prompt for Referring Scenario Comprehension curation.
Figure 13: Quality judge prompt. The judge assesses three criteria: uniqueness, bounding box accuracy, and scenario coherence. It applies a strict conjunction rule: an instance is kept only when all three criteria are satisfied.
Figure 14: RSC training examples.
Figure 15: RSC examples with difficulty level "easy".
Figure 16: RSC examples with difficulty level "medium".
Figure 17: RSC examples with difficulty level "hard".
mIoU (%)
Cat Acc (%)
Tag
Baseline
SFT
GRPO
Baseline
SFT
GRPO
U1
27.24
41.71
46.78
26.24
13.57
22.20
U2
16.98
26.44
31.67
16.52
11.71
20.28
C1
22.85
36.06
39.54
12.32
10.53
19.26
C2
21.30
33.00
39.29
19.93
12.22
21.33
C3
20.87
31.44
36.40
27.82
14.31
22.18
Appendix
Table 11: Tag-level analysis on RSC-OOD. mIoU and category accuracy (Cat Acc) broken down by difficulty tag for the baseline (Qwen2.5-VL), SFT, and GRPO.
Figure 18: Qualitative examples of ScenGround. Green for the GT box and red for the Predicted box.
wiou
wcat
wfmt
wstruct
Stage 1
0.75
0.15
0.07
0.03
Stage 2 (start)
0.55
0.25
0.12
0.08
Stage 2 (late)
0.75
0.20
0.04
0.01
Appendix
Table 12: Reward weight schedules. Stage 2 weights are set by the RewardWeightScheduler from step 0, overriding the initialisation values.
Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.
Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios. OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions. Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC. Project page: https://zongjianwu.github.io/openref
Zongjian Wu, Lei Zhang
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision and language models, yet most prior work focuses on natural images. In contrast, existing chart referring expression grounding-related benchmarks remain limited: (1) they largely adopt bounding boxes, constraining localization precision for fine chart elements (2) they mostly assume a single and two referred target instances, failing to handle multi-instance target references; (3) the language expressions over-rely on textual cues or data-rank clues (4) they cover only a narrow range of chart types. To address these issues, we introduce a chart referring expression grounding benchmark that systematically supports multiple localization forms, multiple referred targets, diverse grounding cues and diverse chart types. Results across representative multimodal large models reveal a significant performance gap. We further introduce a code-driven synthesis pipeline that exploits the inherent alignment between plotting programs and rendered chart primitives to derive pixel accurate instance masks across chart element types and granularities. We train an instance segmentation model with the synthesized masks and integrate it into a general-purpose multimodal grounding framework. The resulting system consistently outperforms baselines on our benchmark and generalizes well to a ChartQA-derived real-chart grounding benchmark.
Tianhao Niu, Ziyu Han, Xuan Dong +2
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China