Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Beijing Normal University · Zhejiang University · University of the Chinese Academy of Sciences · University of Science and Technology Beijing · Shanghai Jiao Tong University
Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text-latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for subsequent grounding. Trajectory-Aware Parallel Decoding then recovers all coordinate blocks in parallel through four geometry-constrained refinement rounds, with each block conditioned on its corresponding trajectory prefix, thereby reducing within-box sequential dependencies and eliminating cross-box autoregression. We further apply Grounding-Aware GRPO to promote consistent grounding across steps and learn adaptive trajectory generation from complex REC queries. ReferFlow achieves leading performance on general REC benchmarks and ReferFlow-Bench.
Figures & tables
Figure 1: ReferFlow-Bench evaluates grounding trajectories, exposing intermediate localization failures hidden by final-only REC evaluation. ReferFlow combines object-indexed region states with parallel geometry-constrained coordinate decoding to support consistent grounding across steps.
Figure 2: Overview of ReferFlow-Bench. (a) Four examples illustrate the trajectory types with different cross-step region dependencies. (b) Distribution of the 10,000 constructed trajectories.
Figure 3: Overview of ReferFlow. ReferFlow builds an interleaved text-latent grounding trajectory with Stateful Region Latent Reasoning, then uses Trajectory-Aware Parallel Decoding to decode all target boxes in parallel from their corresponding trajectory prefixes through geometry-constrained refinement. Grounding-Aware GRPO learns adaptive trajectories for complex REC.
Model
Params.
Overall
Refinement
Shift
Reset
Mixture
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Proprietary MLLMs
Gemini 3.1 Pro ( Google DeepMind, 2026 )
–
33.2
8.9
33.9
25.4
9.4
47.4
41.9
16.6
34.1
34.3
5.4
23.0
31.2
4.3
31.1
GPT-5.6 Sol ( OpenAI, 2026 )
–
44.9
12.7
41.5
33.4
10.9
50.6
51.8
24.0
37.8
47.0
8.4
32.6
47.5
7.4
45.1
Open-Source General MLLMs
Ovis2.5 ( Lu et al., 2025 )
2B
27.6
3.2
25.4
18.2
1.8
26.8
36.7
7.2
23.2
29.6
1.2
27.9
25.9
2.4
23.6
Table 1: Performance of state-of-the-art models on ReferFlow-Bench across four trajectory types. Best results are shown in bold , and the second-best are underlined .
Model
Params.
RefCOCO
RefCOCO+
RefCOCOg
Ref-L4
Ref-Adv
GroundingME
HumanRef
PR-Bench
General MLLMs
Qwen3-VL ( Bai et al., 2025 )
2B
87.4
78.1
84.9
85.4
42.5
26.6
58.9
52.8
Qwen3.5 ( Qwen Team, 2026a )
4B
90.1
83.5
88.4
88.7
59.3
39.6
69.0
62.4
Youtu-VL ( Wei et al., 2026 )
4B
93.2
89.8
92.6
83.3
48.5
14.1
55.3
53.2
InternVL3.5 ( Wang et al., 2025 )
8B
91.9
87.6
89.5
80.9
49.7
11.8
40.0
41.5
GLM-4.6V-Flash ( Team et al., 2025 )
9B
88.1
80.0
88.7
72.7
58.0
29.1
75.2
58.9
Table 2: Comparisons on general referring expression comprehension benchmarks. Best results are shown in bold , and the second-best are underlined . “–” indicates unavailable or unreported results.
Figure 4: Comparison of T-Acc between ordered and shuffled grounding steps.
Figure 5: Comparison of F-Acc across different trajectory lengths.
Figure 6: Comparison of Q-Acc across different types and numbers of transition changes.
Setting
Supervision
RefCOCOg
GroundingME
ReferFlow-Bench
ReferFlow-Bench Q-Acc by Trajectory Length
Spatial
Visual
Q-Acc
T-Acc
F-Acc
2
3
4
5
6+
Text
–
–
87.9
32.4
52.6
17.9
48.2
58.8
54.7
50.1
44.0
41.2
Region Latent
×
×
88.5
32.8
53.0
18.1
48.7
59.6
55.4
49.8
43.9
40.8
Region Latent
✓
×
89.7
34.2
54.4
19.3
50.4
60.8
56.6
52.0
45.3
42.0
Region Latent
×
✓
89.1
35.8
55.4
20.0
52.1
60.3
56.5
53.6
49.1
46.4
SRLR
✓
✓
90.5
37.4
56.9
21.3
53.9
61.8
57.8
55.0
51.0
48.9
Table 3: Ablation studies of SRLR under different spatial and visual supervision settings.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of the annotation pipeline for ReferFlow-Bench and ReferFlow-Train.
Figure 8: Statistics of ReferFlow-Bench. Distributions of (a) mean box area per query, (b) trajectory length, (c) query length, and (d) the number of target regions per query. Query-level statistics are computed over step-level queries.
Dataset
SFT
RL
ReferFlow-Train
400K
3K
RefCOCO/+/g
320K
1K
Rexverse-2M
400K
5K
Visual Genome
450K
2K
HumanRef-CoT-45k
30K
2K
Total
1.6M
13K
Appendix
Table 5: Composition of training data.
1:
for each (t,n) do
2:
Bt,n←Encode(bt,n∗) // three digits per coordinate
Estimate Lbox under the trajectory-aware attention mask (Eq. ( 9 )).
Appendix
Algorithm 1 Training of TAPD
1:
Bt,n←[MASK]12,∀(t,n)
2:
for r=1,…,4 do
3:
Construct H(r) from H and the current {Bt,n} (Eq. ( 5 )).
4:
Compute {pr,c,j} for all blocks in one forward pass using the trajectory-aware attention mask (Eq. ( 6 )).
5:
for each Bt,n in parallel do
6:
Cr←{c:Bt,n[c]=[MASK]3} // B[c] : three digit tokens
7:
Determine Ir(c) from recovered coordinates and box constraints, c∈Cr .
Appendix
Algorithm 2 Inference of TAPD
Model
Size
Refinement
Shift
Attribute
Relational
Spatial
Semantic
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Gemini 3.1 Pro
–
28.5
11.7
50.7
22.4
7.2
44.3
43.1
17.5
35.5
39.6
14.9
31.4
GPT-5.6 Sol
–
36.0
12.8
53.4
30.9
9.1
47.9
52.8
24.7
39.0
49.9
22.7
35.5
Ovis2.5
2B
21.1
3.4
30.0
15.4
0.3
23.8
37.8
8.0
24.5
34.6
5.7
20.7
Qwen3-VL
2B
24.8
5.0
33.5
18.2
0.5
26.5
45.6
13.2
29.7
41.8
10.6
25.6
Appendix
Table 6: Performance comparison on the Refinement and Shift subtypes of ReferFlow-Bench.
Model
Size
Reset
Mixture
Cross-category
Intra-category
Sequential
Interleaved
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Gemini 3.1 Pro
–
36.2
6.6
24.9
30.9
3.2
19.6
34.4
6.6
34.4
27.3
1.5
27.0
GPT-5.6 Sol
–
48.5
9.4
34.2
44.3
6.6
29.7
50.2
9.3
47.9
44.2
5.1
41.6
Ovis2.5
2B
31.4
1.8
29.7
26.3
0.1
24.6
28.9
4.0
26.8
22.2
0.4
19.7
Qwen3-VL
2B
41.8
7.1
32.9
36.5
3.5
27.3
40.2
6.7
32.8
32.8
1.3
24.8
Appendix
Table 7: Performance comparison on the Reset and Mixture subtypes of ReferFlow-Bench.
Figure 9: Comparisons between multi-turn and single-turn evaluation protocols on ReferFlow-Bench.
Figure 10: Effects of intermediate grounding on final localization across trajectory types.
Model
RefCOCOg
GroundingME
ReferFlow-Bench
Time (s)
Len.
Acc.
Time (s)
Len.
Acc.
Time (s)
Len.
Acc.
Qwen3-VL-2B
1.5
29.4
84.3
2.8
85.6
27.6
6.3
237.1
34.8
LocateAnything
0.9 (24.5% fallback)
10.2
88.0
1.9 (66.5% fallback)
53.7
23.9
–
–
–
ReferFlow
0.8
18.8
92.2
1.2
62.4
43.4
1.8
119.4
63.2
Appendix
Table 8: Efficiency analysis of ReferFlow compared with representative models. We report average wall-clock time (Time), sequence length (Len.), and accuracy (Acc.) across three benchmarks. For ReferFlow-Bench, Acc. denotes Q-Acc. Fallback denotes re-decoding unreliable parallel outputs using next-token prediction. – denotes unsupported evaluation settings.
Figure 11: Effect of historical region states on final localization.
Figure 12: Attention visualization of region latents. Heatmaps depict the Gaussian-smoothed aggregation of relevant image patches for region latents.
Figure 14: Proportions of direct and trajectory-based grounding across benchmarks.
Setting
Final Acc.(%)
Trajectory Ratio (%)
Avg. Steps
SFT
39.1
48.0
1.72
RL w/o RTR
40.6
64.0
2.06
RL w/ RTR
42.1
71.0
2.03
Appendix
Table 10: Effects of trajectory reward on GroundingME. (a) Final localization accuracy, trajectory ratio, and average grounding steps, where trajectory ratio denotes the proportion of multi-step grounding outputs. (b) Intermediate grounding accuracy overall and across different trajectory lengths.
Figure 15: Visualization results for Refinement and Shift on ReferFlow-Bench. Each panel presents a challenging example and the corresponding predictions from representative models.
Figure 16: Visualization results for Reset and Mixture on ReferFlow-Bench. Each panel presents a challenging example and the corresponding predictions from representative models.
Figure 17: Visualization results of ReferFlow. Blue dashed boxes denote intermediate reference regions, while green solid boxes denote predicted target regions.
Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.
Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios. OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions. Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC. Project page: https://zongjianwu.github.io/openref
Zongjian Wu, Lei Zhang
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.
Ruozhen He, Nisarg A. Shah, Qihua Dong +3
Rice University · Johns Hopkins University · Northeastern University