Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Beijing Normal University · Zhejiang University · University of the Chinese Academy of Sciences · University of Science and Technology Beijing · Shanghai Jiao Tong University
Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text-latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for subsequent grounding. Trajectory-Aware Parallel Decoding then recovers all coordinate blocks in parallel through four geometry-constrained refinement rounds, with each block conditioned on its corresponding trajectory prefix, thereby reducing within-box sequential dependencies and eliminating cross-box autoregression. We further apply Grounding-Aware GRPO to promote consistent grounding across steps and learn adaptive trajectory generation from complex REC queries. ReferFlow achieves leading performance on general REC benchmarks and ReferFlow-Bench.
Figures & tables
Figure 1: ReferFlow-Bench evaluates grounding trajectories, exposing intermediate localization failures hidden by final-only REC evaluation. ReferFlow combines object-indexed region states with parallel geometry-constrained coordinate decoding to support consistent grounding across steps.
Figure 2: Overview of ReferFlow-Bench. (a) Four examples illustrate the trajectory types with different cross-step region dependencies. (b) Distribution of the 10,000 constructed trajectories.
Figure 3: Overview of ReferFlow. ReferFlow builds an interleaved text-latent grounding trajectory with Stateful Region Latent Reasoning, then uses Trajectory-Aware Parallel Decoding to decode all target boxes in parallel from their corresponding trajectory prefixes through geometry-constrained refinement. Grounding-Aware GRPO learns adaptive trajectories for complex REC.
Model
Params.
Overall
Refinement
Shift
Reset
Mixture
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Proprietary MLLMs
Gemini 3.1 Pro ( Google DeepMind, 2026 )
–
33.2
8.9
33.9
25.4
9.4
47.4
41.9
16.6
34.1
34.3
5.4
23.0
31.2
4.3
31.1
GPT-5.6 Sol ( OpenAI, 2026 )
–
44.9
12.7
41.5
33.4
10.9
50.6
51.8
24.0
37.8
47.0
8.4
32.6
47.5
7.4
45.1
Open-Source General MLLMs
Ovis2.5 ( Lu et al., 2025 )
2B
27.6
3.2
25.4
18.2
1.8
26.8
36.7
7.2
23.2
29.6
1.2
27.9
25.9
2.4
23.6
Table 1: Performance of state-of-the-art models on ReferFlow-Bench across four trajectory types. Best results are shown in bold , and the second-best are underlined .
Model
Params.
RefCOCO
RefCOCO+
RefCOCOg
Ref-L4
Ref-Adv
GroundingME
HumanRef
PR-Bench
General MLLMs
Qwen3-VL ( Bai et al., 2025 )
2B
87.4
78.1
84.9
85.4
42.5
26.6
58.9
52.8
Qwen3.5 ( Qwen Team, 2026a )
4B
90.1
83.5
88.4
88.7
59.3
39.6
69.0
62.4
Youtu-VL ( Wei et al., 2026 )
4B
93.2
89.8
92.6
83.3
48.5
14.1
55.3
53.2
InternVL3.5 ( Wang et al., 2025 )
8B
91.9
87.6
89.5
80.9
49.7
11.8
40.0
41.5
GLM-4.6V-Flash ( Team et al., 2025 )
9B
88.1
80.0
88.7
72.7
58.0
29.1
75.2
58.9
Table 2: Comparisons on general referring expression comprehension benchmarks. Best results are shown in bold , and the second-best are underlined . “–” indicates unavailable or unreported results.
Figure 4: Comparison of T-Acc between ordered and shuffled grounding steps.
Figure 5: Comparison of F-Acc across different trajectory lengths.
Figure 6: Comparison of Q-Acc across different types and numbers of transition changes.
Setting
Supervision
RefCOCOg
GroundingME
ReferFlow-Bench
ReferFlow-Bench Q-Acc by Trajectory Length
Spatial
Visual
Q-Acc
T-Acc
F-Acc
2
3
4
5
6+
Text
–
–
87.9
32.4
52.6
17.9
48.2
58.8
54.7
50.1
44.0
41.2
Region Latent
×
×
88.5
32.8
53.0
18.1
48.7
59.6
55.4
49.8
43.9
40.8
Region Latent
✓
×
89.7
34.2
54.4
19.3
50.4
60.8
56.6
52.0
45.3
42.0
Region Latent
×
✓
89.1
35.8
55.4
20.0
52.1
60.3
56.5
53.6
49.1
46.4
SRLR
✓
✓
90.5
37.4
56.9
21.3
53.9
61.8
57.8
55.0
51.0
48.9
Table 3: Ablation studies of SRLR under different spatial and visual supervision settings.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of the annotation pipeline for ReferFlow-Bench and ReferFlow-Train.
Figure 8: Statistics of ReferFlow-Bench. Distributions of (a) mean box area per query, (b) trajectory length, (c) query length, and (d) the number of target regions per query. Query-level statistics are computed over step-level queries.
Dataset
SFT
RL
ReferFlow-Train
400K
3K
RefCOCO/+/g
320K
1K
Rexverse-2M
400K
5K
Visual Genome
450K
2K
HumanRef-CoT-45k
30K
2K
Total
1.6M
13K
Appendix
Table 5: Composition of training data.
1:
for each (t,n) do
2:
Bt,n←Encode(bt,n∗) // three digits per coordinate
Estimate Lbox under the trajectory-aware attention mask (Eq. ( 9 )).
Appendix
Algorithm 1 Training of TAPD
1:
Bt,n←[MASK]12,∀(t,n)
2:
for r=1,…,4 do
3:
Construct H(r) from H and the current {Bt,n} (Eq. ( 5 )).
4:
Compute {pr,c,j} for all blocks in one forward pass using the trajectory-aware attention mask (Eq. ( 6 )).
5:
for each Bt,n in parallel do
6:
Cr←{c:Bt,n[c]=[MASK]3} // B[c] : three digit tokens
7:
Determine Ir(c) from recovered coordinates and box constraints, c∈Cr .
Appendix
Algorithm 2 Inference of TAPD
Model
Size
Refinement
Shift
Attribute
Relational
Spatial
Semantic
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Gemini 3.1 Pro
–
28.5
11.7
50.7
22.4
7.2
44.3
43.1
17.5
35.5
39.6
14.9
31.4
GPT-5.6 Sol
–
36.0
12.8
53.4
30.9
9.1
47.9
52.8
24.7
39.0
49.9
22.7
35.5
Ovis2.5
2B
21.1
3.4
30.0
15.4
0.3
23.8
37.8
8.0
24.5
34.6
5.7
20.7
Qwen3-VL
2B
24.8
5.0
33.5
18.2
0.5
26.5
45.6
13.2
29.7
41.8
10.6
25.6
Appendix
Table 6: Performance comparison on the Refinement and Shift subtypes of ReferFlow-Bench.
Model
Size
Reset
Mixture
Cross-category
Intra-category
Sequential
Interleaved
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Q-Acc
T-Acc
F-Acc
Gemini 3.1 Pro
–
36.2
6.6
24.9
30.9
3.2
19.6
34.4
6.6
34.4
27.3
1.5
27.0
GPT-5.6 Sol
–
48.5
9.4
34.2
44.3
6.6
29.7
50.2
9.3
47.9
44.2
5.1
41.6
Ovis2.5
2B
31.4
1.8
29.7
26.3
0.1
24.6
28.9
4.0
26.8
22.2
0.4
19.7
Qwen3-VL
2B
41.8
7.1
32.9
36.5
3.5
27.3
40.2
6.7
32.8
32.8
1.3
24.8
Appendix
Table 7: Performance comparison on the Reset and Mixture subtypes of ReferFlow-Bench.
Figure 9: Comparisons between multi-turn and single-turn evaluation protocols on ReferFlow-Bench.
Figure 10: Effects of intermediate grounding on final localization across trajectory types.
Model
RefCOCOg
GroundingME
ReferFlow-Bench
Time (s)
Len.
Acc.
Time (s)
Len.
Acc.
Time (s)
Len.
Acc.
Qwen3-VL-2B
1.5
29.4
84.3
2.8
85.6
27.6
6.3
237.1
34.8
LocateAnything
0.9 (24.5% fallback)
10.2
88.0
1.9 (66.5% fallback)
53.7
23.9
–
–
–
ReferFlow
0.8
18.8
92.2
1.2
62.4
43.4
1.8
119.4
63.2
Appendix
Table 8: Efficiency analysis of ReferFlow compared with representative models. We report average wall-clock time (Time), sequence length (Len.), and accuracy (Acc.) across three benchmarks. For ReferFlow-Bench, Acc. denotes Q-Acc. Fallback denotes re-decoding unreliable parallel outputs using next-token prediction. – denotes unsupported evaluation settings.
Figure 11: Effect of historical region states on final localization.
Figure 12: Attention visualization of region latents. Heatmaps depict the Gaussian-smoothed aggregation of relevant image patches for region latents.
Figure 14: Proportions of direct and trajectory-based grounding across benchmarks.
Setting
Final Acc.(%)
Trajectory Ratio (%)
Avg. Steps
SFT
39.1
48.0
1.72
RL w/o RTR
40.6
64.0
2.06
RL w/ RTR
42.1
71.0
2.03
Appendix
Table 10: Effects of trajectory reward on GroundingME. (a) Final localization accuracy, trajectory ratio, and average grounding steps, where trajectory ratio denotes the proportion of multi-step grounding outputs. (b) Intermediate grounding accuracy overall and across different trajectory lengths.
Figure 15: Visualization results for Refinement and Shift on ReferFlow-Bench. Each panel presents a challenging example and the corresponding predictions from representative models.
Figure 16: Visualization results for Reset and Mixture on ReferFlow-Bench. Each panel presents a challenging example and the corresponding predictions from representative models.
Figure 17: Visualization results of ReferFlow. Blue dashed boxes denote intermediate reference regions, while green solid boxes denote predicted target regions.