Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation
Authors: Alaa Dalaq, Muzammil Behzad
Organizations: King Fahd University of Petroleum and Mineral, Dhahran, 31261, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Dhahran, 31261, Saudi Arabia
Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression. Although pretrained vision-language models have improved semantic grounding, many existing methods still rely on uniform refinement strategies that do not fully match the diverse reasoning requirements of referring expressions. Because of this mismatch, predictions often contain fragmented regions, inaccurate boundaries, or even the wrong object, especially when pretrained backbones are frozen for computational efficiency. To address these limitations, we propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation. SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework. First, we design SERA-Adapter, which inserts an expression-conditioned adapter into selected backbone blocks to improve spatial coherence and boundary precision through expert-guided refinement and cross-modal attention. We then introduce SERA-Fusion, which strengthens intermediate visual representations by reshaping token features into spatial grids and applying geometry-preserving expert transformations before multimodal interaction. In addition, a lightweight routing mechanism adaptively weights expert contributions while remaining compatible with pretrained representations. To make this routing stable under frozen encoders, SERA uses a parameter-efficient tuning strategy that updates only normalization and bias terms, affecting less than 1% of the backbone parameters. Experiments on standard referring image segmentation benchmarks show that SERA consistently outperforms strong baselines, with especially clear gains on expressions that require accurate spatial localization and precise boundary delineation.
Figures & tables
Figure 1: Overview of the proposed SERA. The left side of the figure shows visual-textual examples that illustrate typical challenges in referring image segmentation, along with grammatically challenging prompts that involve fine-grained attribute reasoning, relational grounding, and complex object boundaries. The middle part of the figure shows the SERA architecture, in which a router dynamically assigns routing weights to multiple experts within an expert-guided refinement module. The right side of the figure shows qualitative segmentation results produced by SERA, demonstrating improved alignment between referring expressions and predicted object regions. Ground-truth masks are shown at the bottom for reference.
Figure 2: Architecture of SERA for referring image segmentation. Visual features from a frozen DINOv2 backbone are refined using SERA-Adapter and fused with CLIP text embeddings via SERA-Fusion before decoding the final segmentation mask.
Figure 3: Detailed architecture of the proposed SERA-Adapter. Visual tokens from the backbone are projected into a spatial grid and processed by multi-scale convolutional branches to enrich local context. The resulting features are refined by two specialized experts, a spatial expert and a boundary expert, whose outputs are adaptively weighted by a lightweight router. The refined visual features are then aligned with text representations through cross-modal attention and incorporated into the backbone via a residual update.
Figure 4: Architecture of the proposed SERA-Fusion module. Intermediate visual feature maps are refined using a set of specialized experts targeting complementary cues, including spatial layout, boundary structure, contextual interaction, and global shape consistency. A lightweight routing network computes input-dependent expert weights from pooled spatial features and aggregates the selected expert outputs through weighted fusion. The refined representation preserves spatial resolution and feeds into the segmentation decoder.
Figure 5: Qualitative comparison on RefCOCO. Each row corresponds to a referring expression, with columns showing the original image, ground-truth mask, DETRIS prediction, and SERA (ours) prediction.
Figure 6: Qualitative results on the RefCOCO dataset. From top to bottom: input images with referring expressions, ground-truth masks, and our predicted segmentation masks. IoU values are shown below each prediction. The clickable sample IDs correspond to those listed in Appendix A .
Figure 7: Qualitative results on the RefCOCO+ dataset. From top to bottom: input images with referring expressions, ground-truth masks, and our predicted segmentation masks. IoU scores are reported below each prediction. The clickable sample IDs correspond to Appendix A .
Figure 8: Qualitative results on the RefCOCOg dataset. From top to bottom: input images with referring expressions, ground-truth masks, and our predicted segmentation masks. IoU scores are shown below each prediction. The clickable sample IDs correspond to Appendix A .
Method
RefCOCO
RefCOCO+
G-Ref(g)
(Baseline)
74.90
68.70
65.10
+ SERA-Adapter
75.35 (0.45 ↑ )
69.42 (0.72 ↑ )
65.74 (0.64 ↑ )
+ SERA-Adapter + SERA-Fusion (SERA)
76.50 (1.60 ↑ )
70.40 (1.70 ↑ )
66.62 (1.52 ↑ )
Table 2: Ablation study of SERA components in terms of mIoU. Arrows indicate changes in performance relative to the baseline.
Top- K
val
testA
testB
mIoU
oIoU
mIoU
oIoU
mIoU
oIoU
K=1
75.46
73.32
77.72
76.31
72.80
69.15
K=2
76.47 (1.01 ↑ )
74.65 (1.33 ↑ )
78.17 (0.45 ↑ )
76.91 (0.60 ↑ )
73.55 (0.75 ↑ )
71.23 (2.08 ↑ )
K=3
76.20 (0.74 ↑ )
74.10 (0.78 ↑ )
78.10 (0.38 ↑ )
77.09 (0.78 ↑ )
73.73 (0.93 ↑ )
71.13 (1.98 ↑ )
K=4
76.50 (1.04 ↑ )
74.74 (1.42 ↑ )
78.20 (0.48 ↑ )
77.09 (0.78 ↑ )
73.73 (0.93 ↑ )
71.32 (2.17 ↑ )
Table 3: Top-K routing ablation on RefCOCO. Results are reported in terms of mIoU and oIoU on the validation, testA, and testB splits. Arrows indicate performance change relative to K=1 .
Figure 9: Average routing weight assigned to each expert in SERA-Fusion across routing configurations ( K=1,2,3,4 ) for RefCOCO, RefCOCO+, RefCOCOg (U), and RefCOCOg (G). Boundary and Shape experts consistently receive the highest routing mass, while the Context expert is minimally utilized, indicating stable expert specialization across datasets.
Figure 10: Qualitative comparison between the base model and SERA under different Top- K expert routing settings. Columns correspond to different referring expressions, while rows illustrate the effect of increasing expert capacity. Compared with the base model, SERA produces more complete masks, sharper boundaries, and better foreground-background separation as K increases.
Figure 11: Zero-shot qualitative overlay results on RefCOCOg and RefCOCO+. Numbers above images indicate clickable sample IDs referenced in Appendix A . Values below each image denote IoU scores.
Figure 12: Zero-shot qualitative overlay results on RefCOCO and RefCOCOg. Numbers above images indicate clickable sample IDs referenced in Appendix A . Values below each image denote IoU scores.
Figure 13: Zero-shot qualitative overlay results on RefCOCO and RefCOCOg. Numbers above images indicate clickable sample IDs referenced in Appendix A . Values below each image denote IoU scores.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
ID
Referring Expression
1
catcher
2
man on right
3
dude with jacket
4
far left bowl
5
black blurred car on left
6
horse on left
Appendix
Table 6: Mapping between column indices in different Figures and their corresponding referring expressions. The prompts (with grammatical and spelling mistakes) are taken as it is from the corresponding datasets.
Referring image segmentation aims to localize and segment a target object in an image based on a free-form referring expression. The core challenge lies in effectively bridging linguistic descriptions with object-level visual representations, especially when referring expressions involve detailed attributes and complex inter-object relationships. Existing methods either rely on cross-modal alignment or employ Semantic Segmentation Prompts, but they often lack explicit reasoning mechanisms for grounding language descriptions to target regions in the image. To address these limitations, we propose PPCR, a Progressive Prompt-guided Cross-modal Reasoning framework for referring image segmentation. PPCR explicitly structures the reasoning process as a Semantic Understanding-Spatial Grounding-Instance Segmentation pipeline. Specifically, PPCR first employs multimodal large language models (MLLMs) to generate Semantic Segmentation Prompt that capture key semantic cues of the target object. Based on this semantic context, Spatial Segmentation Prompt are further generated to reason about object location and spatial extent, enabling a progressive transition from semantic understanding to spatial grounding. The Semantic and Spatial Segmentation prompts are then jointly integrated into the segmentation module to guide accurate target localization and segmentation. Extensive experiments on standard referring image segmentation benchmarks demonstrate that PPCR consistently outperforms existing methods. The code will be publicly released to facilitate reproducibility.
Jiachen Li, Hongyun Wang, Jinyu Xu +5
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · University of Electronic Science and Technology of China, Chengdu, China +1
Referring expression segmentation requires language conditioned localization and pixel-accurate masks, but monolithic models can be costly to deploy. We present VespaSeg, a modular pipeline that grounds a text query with a compact vision-language model and converts the predicted box to a mask with MobileSAM. We study Florence-2-base, Florence-2-large, and Moondream2 grounders together with targeted adaptation of the grounding and segmentation stages. Under a repository-specific RefCOCO validation protocol containing the first expression for each of 3,811 referenced-object records, the adapted Florence-2-base pipeline obtains 73.64 mean intersection over union (mIoU) and 84.60 precision at IoU 0.5. On an NVIDIA RTX 6000 Ada GPU it processes 22.8 cached-image queries per second with 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for Florence-2-base and 72.82 for Florence-2-large, while the base model is 1.70 times faster and uses 1.17 GB less allocated memory. Ablations show that ground-truth-box adaptation raises MobileSAM mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. These results support compact, modular grounding and segmentation, while also exposing the need for evaluation on the complete standard RefCOCO expression splits and deployment hardware.
Savindu Dilshan Wickramasinghe
University of Moratuwa · Department of Electronic and Telecommunication Engineering, University of Moratuwa Moratuwa, Sri Lanka
Open-world referring segmentation requires grounding unconstrained language expressions to precise pixel-level regions. Existing multimodal large language models (MLLMs) exhibit strong open-world visual grounding, but their outputs remain limited to sparse bounding-box coordinates and are insufficient for dense visual prediction. Recent MLLM-based segmentation methods either directly predict sparse contour coordinates, struggling to reconstruct continuous object boundaries, or rely on external segmentation foundation models such as the Segment Anything Model (SAM), introducing substantial architectural and deployment overhead. We present Qwen3-VL-Seg, a parameter-efficient framework that treats the MLLM-predicted box as a semantically grounded structural prior and decodes it into pixel-level referring segmentation. At its core, a lightweight box-guided mask decoder combines multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement, introducing only 17M parameters (about 0.4% of the base model). For scalable open-world training, we construct SA1B-ORS, an SA-1B-derived dataset with two subsets: SA1B-CoRS (category-oriented samples) and SA1B-DeRS (descriptive, instance-specific samples). For evaluation, we curate ORS-Bench, a manually screened benchmark with in-distribution and out-of-distribution subsets covering diverse referring expression types. Extensive experiments on referring expression segmentation, visual grounding, and ORS-Bench show that Qwen3-VL-Seg performs strongly across closed-set and open-world settings, with clear advantages on language-intensive instructions and strong out-of-distribution generalization. Evaluations on general multimodal benchmarks further show that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.