cs.CVApr 2, 2026
SaveBeyond Referring Expressions: Scenario Comprehension Visual Grounding
Organizations: Rice University · Johns Hopkins University · Northeastern University
Abstract
Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.
Figures & tables
Figure 1: Referring Scenario Comprehension (RSC) vs. traditional referring expression comprehension (REC). Each row shows the same target object under both paradigms. Traditional REC queries often name the target category directly, allowing success via lexical matching. RSC instead pairs each image with a lengthy scenario-based query specifying a user role, goal, and multiple disambiguating cues, including explicit contrasts against competing objects, and requires output identifying both the target object and its bounding box. The RSC difficulty tags (U/C/S/O/P: Uniqueness, Clutter, Size, Overlap, Position) characterize each instance, enabling fine-grained training and evaluation.
| Benchmark | Train | ID Test | OOD Test | Referring Style | Query Style | Avg. | Reasoning Traces | Difficulty Tags | Competing Mentions |
|---|---|---|---|---|---|---|---|---|---|
| RefCOCO Kazemzadeh et al. (2014) | 120,624 | 10,752 | ✗ | Literal | Phrase | 3.5 | ✗ | ✗ | ✗ |
| RefCOCO+ Kazemzadeh et al. (2014) | 120,191 | 10,615 | ✗ | Literal | Phrase | 3.5 | ✗ | ✗ | ✗ |
| RefCOCOg Mao et al. (2016) | 80,512 | 9,602 | ✗ | Descriptive literal | Phrase | 8.2 | ✗ | ✗ | ✓ |
| Cops-Ref Chen et al. (2020) | 119,603 | 12,586 | ✗ | Compositional literal | Template | 14.4 | ✗ | ✗ | ✓ |
| Ref-Reasoning Yang et al. (2020) | 721,164 | 34,609 | ✗ | Literal | Template | 8.5 | Graph | Graph layout | ✓ |
| SK-VG Chen et al. (2023b) | 23,403 | 6,597 | ✗ | Scene knowledge Q&A | Q&A | 5.3 | ✗ | E/M/H | ✓ |
Table 1: Comparison of referring expression grounding benchmarks. We compare existing datasets with RSC across dataset scale, evaluation splits, referring/query style, reasoning supervision, difficulty annotations, and competing-object mentions. Most prior datasets rely on short literal phrases and lack explicit reasoning traces or OOD evaluation. RSC introduces scenario-based queries written as natural language paragraphs and provides per-instance reasoning trace annotations. It further labels five interpretable difficulty factors (U/C/S/O/P) and includes both in-distribution (ID) and out-of-distribution (OOD) category test sets to evaluate reasoning and generalization.
Figure 2: Phase 1 filters and balances source instances computing five interpretable difficulty tags to form a tag-balanced candidate pool. Phase 2 generates annotations via a two-stage process: a small-scale refinement loop first validates the generation prompt through iterative system refinement and human audit, before large-scale scenario generation is applied to the full candidate pool. Each instance is annotated with a target object description, a scenario query with reasoning traces, acceptable category aliases, and an LLM-predicted bounding box for quality filtering. Phase 3 applies automatic and human quality control: a quality judge filters rough annotations, human auditors verify a stratified sample. The final RSC dataset provides, per instance, a scenario query, reasoning traces, acceptable names, ground-truth box, and difficulty tags.
| Split | Initial | GT Filter | Kept | Quality Filter | Kept |
|---|---|---|---|---|---|
| Train (SFT) | 30,000 | 29,551 | 98.5% | 23,802 | 80.5% |
| Train (RL) | 10,000 | 9,884 | 98.8% | 7,540 | 76.3% |
| Test (ID) | 5,000 | 4,915 | 98.3% | 4,038 | 82.2% |
| Test (OOD) | 5,000 | 3,983 | 79.7% | 3,247 | 81.5% |
Table 2: RSC data retention across quality control stages. After LLM GT Filter : instances passing the automatic schema, box IoU, alias consistency, and leakage gates. After Quality Filter : instances passing the full judge-and-audit pipeline.
Figure 3: ScenGround prompt and output schema. Given an image and a user-driven scenario, the model is instructed to reason inside <think> and to emit a structured JSON inside <answer> . The JSON contains target_object and bbox [x,y,w,h] . Scenarios avoid category names and force disambiguation via attributes, relations, and spatial cues, while <think> must justify the selection and ignore distractors. This schema is used in both TP-SFT and IC-GRPO. To increase the robustness of ScenGround, IC-GRPO uses 7 other templates.
| RSC-ID | RSC-OOD | |||||||
| Model | mIoU | [email protected] | [email protected] | Cat Acc | mIoU | [email protected] | [email protected] | Cat Acc |
| GPT-4o Hurst et al. (2024) | 19.41 | 13.23 | 5.37 | 79.45 | 16.57 | 9.55 | 3.08 | 62.00 |
| Claude 3.7 Anthropic (2025) | 16.64 | 8.32 | 3.71 | 89.67 | 12.04 | 5.54 | 1.87 | 58.98 |
| Grounding DINO ( cat token ) ‡ | 44.60 | 47.55 | 42.03 | — | 32.18 | 31.99 | 27.89 | — |
| Grounding DINO ( ref. cue ) ‡ | 48.99 | 51.84 | 46.02 | — | 38.12 | 38.26 | 34.07 | — |
| InternVL2.5 8B Chen et al. (2024a) | 16.76 | 11.88 | 6.74 | 81.70 | 8.08 | 3.64 | 1.61 | 36.50 |
Table 3: Performance on RSC (ID and OOD). Metrics: mIoU and (higher is better); Cat Acc = category accuracy. ‡ Oracle settings, not directly comparable: Grounding DINO receives privileged inputs unavailable at inference: cat token provides the gold category name directly and ref. cue feeds the conciser reasoning trace conclusion "referring expression cue" (See Figure 2 ) as the open-vocabulary query.
| RSC-ID | RSC-OOD | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | mIoU | [email protected] | [email protected] | Cat Acc | mIoU | [email protected] | [email protected] | Cat Acc |
| Qwen2.5-VL 7B Bai et al. (2025a) | 30.31 | 27.42 | 15.66 | 30.86 | 21.54 | 15.88 | 9.19 | 20.82 |
| SFT | 55.01 | 60.51 | 42.59 | 89.05 | 33.20 | 34.03 | 20.36 | 12.53 |
| GRPO, Single Stage | 54.04 | 59.04 | 38.43 | 94.90 | 37.66 | 36.99 | 20.82 | 20.05 |
| GRPO, Stage 1 | 55.68 | 60.95 | 41.88 | 94.04 | 36.93 | 36.59 | 21.37 | 19.00 |
| GRPO, Stage 2 | 55.68 | 60.90 | 42.32 | 94.23 | 38.37 | 38.11 | 22.64 | 21.13 |
Table 4: Ablation of curriculum in IC-GRPO on RSC in-domain (RSC-ID) and out-of-domain (RSC-OOD). Metrics: mIoU and for boxes; Cat Acc for category naming. Single Stage trains on the union of all RL samples without curriculum; Stage- trains on the stage- slice, all other settings unchanged.
| RefCOCO+ | RefCOCOg | ||||
|---|---|---|---|---|---|
| Method | Val | Test A | Test B | Val | Test |
| Qwen2.5 - VL | 84.20 | 89.10 | 76.90 | 87.20 | 87.20 |
| Qwen2.5 - VL ∗ | 52.54 | 56.75 | 52.53 | 52.46 | 51.40 |
| ScenGround ∗ , SFT | 50.86 | 43.84 | 50.74 | 65.45 | 63.39 |
| ScenGround ∗ , GRPO | 70.16 | 74.63 | 70.05 | 78.19 | 75.61 |
Table 5: Referring expression comprehension on RefCOCO+ and RefCOCOg. Metric: [email protected] (%, ). † Use the official grounding pipeline and prompt Bai et al. (2025a) . ∗ Use a custom prompt.
| RSC-ID | RSC-OOD | |||||
|---|---|---|---|---|---|---|
| Method | mIoU | Acc @0.5 | Acc @0.7 | mIoU | Acc @0.5 | Acc @0.7 |
| Qwen2.5-VL 7B | 27.37 | 24.09 | 13.42 | 19.11 | 13.50 | 7.70 |
| ScenGround, SFT | 51.18 | 55.80 | 37.99 | 40.71 | 42.65 | 25.86 |
| ScenGround, GRPO | 50.90 | 54.38 | 37.01 | 40.11 | 40.30 | 24.50 |
Table 6: Localization under the box-only setting on RSC in-distribution (RSC-ID) and out-of-distribution (RSC-OOD). Metrics are mIoU and .
| Text Only | Text-Image | Standard | ||||
|---|---|---|---|---|---|---|
| Method | ID | OOD | ID | OOD | ID | OOD |
| Qwen2.5-VL 7B | 56.24 | 22.34 | 77.54 | 33.33 | 29.65 | 19.28 |
| ScenGround-SFT | 81.03 | 10.69 | 93.66 | 14.92 | 92.72 | 12.30 |
| ScenGround-GRPO | 85.13 | 20.22 | 94.73 | 22.68 | 93.71 | 20.67 |
Table 7: Category accuracy on RSC in-distribution (ID) and out-of-distribution (OOD) under three input–output settings. Text Only : predict category from scenario only. Text-Image : predict category from scenario+image (no box). Standard : full task.
Figure 4: Qualitative results on RSC. Red box for ground truth and blue box for ScenGround prediction.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Query length distribution. RSC queries are substantially longer than RefCOCO+ and RefCOCOg expressions, reflecting the paragraph-length scenario descriptions that specify user roles, goals, and disambiguating cues. RefCOCO+ and RefCOCOg peak below 10 words; RSC peaks around 50–60 words.
Figure 6: Instance size distribution. RSC covers a broader range of instance sizes than RefCOCO+ and RefCOCOg, with notably higher density at smaller instances.
| Dataset | Unique Vocabulary |
|---|---|
| RefCOCO+ | 4,240 |
| RefCOCOg | 4,917 |
| RSC (Ours) | 9,086 |
Table 8: Vocabulary size comparison. Number of unique words across all queries in each dataset’s test split.
Figure 7: Difficulty tag distributions for RSC-ID and RSC-OOD. Each panel shows the per-tag marginal distribution for the in-distribution (RSC-ID) and out-of-distribution (RSC-OOD) test splits across all five axes: Uniqueness (U), Clutter (C), Size (S), Overlap (O), and Position (P).
Figure 8: Top-10 words per part-of-speech category in RSC scenarios. Frequency counts of the most common nouns, verbs, adverbs, and prepositions across all RSC scenario queries.
Figure 9: Top 20 most frequent categories in RSC-ID and RSC-OOD test splits.
Figure 10: RSC-ID category frequency distribution. Frequency of all 79 COCO categories across 4,038 in-distribution test instances, plotted on a log scale. The distribution is moderately long-tailed, with common household objects dominating.
Figure 11: RSC-OOD category frequency distribution. Frequency of the top 80 most common categories among 395 LVIS categories across 3,247 out-of-distribution test instances, plotted on a log scale. The long-tailed distribution reflects LVIS’s diverse fine-grained vocabulary.
| LLM-generated | Human-authored | |||
|---|---|---|---|---|
| Method | mIoU | Cat. Acc | mIoU | Cat. Acc |
| Qwen2.5-VL | 33.4 | 31.0 | 31.3 | 32.0 |
| ScenGround | 58.0 | 96.0 | 59.2 | 98.0 |
Table 9: Model performance on human-authored vs. LLM-generated scenarios for the same 100 RSC instances. The human-authored scenario query is randomly drawn from one of the three annotators. ScenGround performs comparably on human-authored queries, supporting ecological validity and confirming that training does not overfit to LLM-generated patterns.
| Human-authored | LLM-generated |
|---|---|
| There is a tourist walking down the street who needs to cross at the intersection. You are looking for a green light on a pole, away from the main intersection light. It’s closer to the left side of the street and looks higher than the building. | As a pedestrian waiting to cross, you want to check if it’s safe. You look for the green-lit signal on the rightmost pole, above a blue sign, away from the main intersection lights. Its color, position, and separation from the cluster help you identify it. |
| A parent is watching from the beach, making sure the kid with dark hair is safe. You notice most of his body is submerged, he’s close to an adult, and he’s the leftmost in the group. Please identify the kid in the crowd. | As a parent watching from the shore, you want to check if your child, who has dark hair and is the leftmost in the group, is safe. They are closest to the water’s edge, mostly submerged, and positioned to the left of the shirtless swimmer. |
| A student is organizing the desk. They are looking for a green-handled item that sits on the desk. There are orange scissors and colorful pens in a mug, but those are not needed. Find the target object to the left of the mug that can be used to cut things. | As a teacher organizing supplies, you need the small green-handled item lying on the desk, not the larger orange-handled ones in the mug. Its bright green color, flat position, and separation from the mug make it easy to spot. |
Table 10: Human-authored vs. LLM-generated scenario pairs for the same targets. Both convey equivalent referential content through different surface forms.
Figure 12: The prompt for Referring Scenario Comprehension curation.
Figure 13: Quality judge prompt. The judge assesses three criteria: uniqueness, bounding box accuracy, and scenario coherence. It applies a strict conjunction rule: an instance is kept only when all three criteria are satisfied.
Figure 14: RSC training examples.
Figure 15: RSC examples with difficulty level "easy".
Figure 16: RSC examples with difficulty level "medium".
Figure 17: RSC examples with difficulty level "hard".
| mIoU (%) | Cat Acc (%) | |||||
| Tag | Baseline | SFT | GRPO | Baseline | SFT | GRPO |
| U1 | 27.24 | 41.71 | 46.78 | 26.24 | 13.57 | 22.20 |
| U2 | 16.98 | 26.44 | 31.67 | 16.52 | 11.71 | 20.28 |
| C1 | 22.85 | 36.06 | 39.54 | 12.32 | 10.53 | 19.26 |
| C2 | 21.30 | 33.00 | 39.29 | 19.93 | 12.22 | 21.33 |
| C3 | 20.87 | 31.44 | 36.40 | 27.82 | 14.31 | 22.18 |
Table 11: Tag-level analysis on RSC-OOD. mIoU and category accuracy (Cat Acc) broken down by difficulty tag for the baseline (Qwen2.5-VL), SFT, and GRPO.
Figure 18: Qualitative examples of ScenGround. Green for the GT box and red for the Predicted box.
| Stage 1 | 0.75 | 0.15 | 0.07 | 0.03 |
|---|---|---|---|---|
| Stage 2 (start) | 0.55 | 0.25 | 0.12 | 0.08 |
| Stage 2 (late) | 0.75 | 0.20 | 0.04 | 0.01 |
Table 12: Reward weight schedules. Stage 2 weights are set by the RewardWeightScheduler from step 0, overriding the initialisation values.
More in this topic
Vision-Language Grounding
Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.
See all 418 papers