Organizations: School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · University of Electronic Science and Technology of China, Chengdu, China · Wuhan Vocational College of Software and Engineering, Wuhan, China
Referring image segmentation aims to localize and segment a target object in an image based on a free-form referring expression. The core challenge lies in effectively bridging linguistic descriptions with object-level visual representations, especially when referring expressions involve detailed attributes and complex inter-object relationships. Existing methods either rely on cross-modal alignment or employ Semantic Segmentation Prompts, but they often lack explicit reasoning mechanisms for grounding language descriptions to target regions in the image. To address these limitations, we propose PPCR, a Progressive Prompt-guided Cross-modal Reasoning framework for referring image segmentation. PPCR explicitly structures the reasoning process as a Semantic Understanding-Spatial Grounding-Instance Segmentation pipeline. Specifically, PPCR first employs multimodal large language models (MLLMs) to generate Semantic Segmentation Prompt that capture key semantic cues of the target object. Based on this semantic context, Spatial Segmentation Prompt are further generated to reason about object location and spatial extent, enabling a progressive transition from semantic understanding to spatial grounding. The Semantic and Spatial Segmentation prompts are then jointly integrated into the segmentation module to guide accurate target localization and segmentation. Extensive experiments on standard referring image segmentation benchmarks demonstrate that PPCR consistently outperforms existing methods. The code will be publicly released to facilitate reproducibility.
Figures & tables
Figure 1: An example of referring image segmentation, where a free-form referring expression contains detailed attributes and complex relationships among multiple entities (e.g., bottle of water , guy , and yellow short ), requiring relational reasoning to accurately localize the target region.
Figure 2: Illustration of different referring image segmentation paradigms. (a) Cross-modal alignment methods perform matching by directly aligning image and text representations. (b) MLLMs-based methods generate Semantic Segmentation Prompt to guide segmentation, but lack explicit spatial grounding. (c) Our PPCR progressively generates Semantic and Spatial Segmentation Prompt, explicitly bridging semantic understanding and spatial grounding for accurate segmentation.
Figure 3: Overview of PPCR for referring image segmentation. The framework follows a Semantic Understanding-Spatial Grounding-Instance Segmentation pipeline. Given an image and a referring expression, a multimodal large language model first generates a Semantic Segmentation Prompt (Semantic SP) via the LoRA A branch, which then guides the generation of a Spatial Segmentation Prompt (Spatial SP) via the LoRA B branch. The Spatial Segmentation Prompt is mapped to bounding box coordinates, and is jointly used with the Semantic Segmentation Prompt by SAM for instance segmentation. Purple and orange arrows denote semantic and spatial information flows.
Figure 4: Illustration of the Semantic Prompt Template used in the Semantic Understanding stage.
Figure 5: Illustration of the Spatial Prompt Template used in the Spatial Grounding stage.
Method
MLLMs
RefCOCO
RefCOCO+
RefCOCOg
oIoU
mIoU
oIoU
mIoU
oIoU
mIoU
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val(U)
Test(U)
Val(U)
Test(U)
CRIS † [ 7 ]
–
–
–
–
70.47
73.18
66.10
–
–
–
62.27
68.08
53.68
–
–
59.69
60.01
LAVT [ 8 ]
–
72.73
75.82
68.79
74.76
76.89
70.94
62.14
68.38
55.10
65.81
70.97
59.23
61.23
62.09
63.34
63.63
CoupAlign [ 20 ]
–
74.70
77.76
70.58
75.49
77.15
63.98
62.92
68.34
56.69
66.17
71.56
60.23
62.84
62.22
63.76
64.32
CARIS † [ 9 ]
–
76.63
79.40
73.52
–
–
–
68.03
73.70
60.41
–
–
–
67.95
69.75
–
–
Table 1: Comparison of referring image segmentation methods on RefCOCO, RefCOCO+, and RefCOCOg. †means the results are excerpted from the original paper.
Variant
RefCOCO
RefCOCO+
RefCOCOg
oIoU
mIoU
oIoU
mIoU
oIoU
mIoU
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val(U)
Test(U)
Val(U)
Test(U)
w/o Spatial SP
79.50
81.92
76.90
80.76
82.78
78.36
72.10
76.60
64.60
73.76
78.05
68.38
73.66
74.70
75.11
75.66
w/ Spatial SP (RE)
80.22
82.76
76.66
80.99
82.98
77.96
70.93
78.04
64.25
73.35
78.81
67.45
74.30
75.08
75.17
75.73
w/ Spatial SP (RE + Ptxtseg )
79.80
82.32
77.08
80.54
82.59
78.34
72.15
77.50
66.08
73.72
78.44
69.15
74.63
75.10
75.30
75.74
PPCR
80.10
83.55
77.60
81.10
83.69
78.33
72.50
78.52
65.55
74.34
79.56
69.33
74.92
75.70
75.94
76.10
Table 2: Ablation study on different reasoning mechanisms on RefCOCO, RefCOCO+, and RefCOCOg.
Variant
RefCOCO
RefCOCO+
RefCOCOg
oIoU
mIoU
oIoU
mIoU
oIoU
mIoU
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val
TestA
TestB
Val(U)
Test(U)
Val(U)
Test(U)
w/ Attn-based Box Prediction
80.11
82.31
76.81
81.09
82.98
77.92
71.87
76.34
65.20
73.93
78.33
68.44
75.37
75.03
75.97
75.90
w/ GT Box (Oracle)
88.32
88.93
86.84
87.79
88.12
86.72
87.94
88.76
87.12
87.58
87.95
86.82
87.45
87.58
85.58
85.73
PPCR
80.10
83.55
77.60
81.10
83.69
78.33
72.50
78.52
65.55
74.34
79.56
69.33
74.92
75.70
75.94
76.10
Table 3: Ablation study on Spatial Grounding on RefCOCO, RefCOCO+, and RefCOCOg.
Figure 6: Qualitative comparisons on RefCOCOg. From left to right: Ground-Truth masks (GT), GLaMM results, PPCR results, the Spatial Segmentation Prompt Ground Truth (Spatial SP GT), and the Spatial Segmentation Prompt (Spatial SP).
King Fahd University of Petroleum and Mineral, Dhahran, 31261, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Dhahran, 31261, Saudi Arabia
School of Computer Science and Engineering, Sun Yat-sen University, China · Guangdong Province Key Laboratory of Information Security Technology, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China