Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China · National Engineering Research Center of Visual Technology, Peking University, Beijing, China · Nanjing University of Information Science and Technology, China · Department of Electrical and Electronic Engineering, Yonsei University, Korea
Amodal segmentation aims to predict complete object masks covering both visible and occluded regions, serving as a critical foundation for robust multimedia scene understanding and interactive media editing. This task poses significant challenges in complex visual media due to severe occlusions and extreme shape variation, ranging from rigid furniture to highly deformable clothing. Existing one-size-fits-all approaches struggle to reason about diverse amodal shapes in multimedia content due to limited representation capacity. While a mixture-of-experts framework is a natural solution, naively applying it without considering the underlying shape distribution leads to mismatched routing and redundant experts. To address these issues, we introduce ShapeMoE, a shape-specific sparse mixture-of-experts framework designed for complex visual scene parsing. The key idea is to learn a latent shape distribution space and dynamically route each object to a lightweight expert tailored to its geometric characteristics. Specifically, ShapeMoE encodes each object into a compact Gaussian embedding that captures key shape features. A Shape-Aware Sparse Router then maps the object to the most suitable expert, enabling precise shape-aware routing. Each expert is lightweight and specializes in recovering occluded regions for specific shape patterns. ShapeMoE is designed for shape-specific expert specialization while maintaining computational efficiency. Extensive experiments on three challenging benchmarks demonstrate that ShapeMoE consistently outperforms state-of-the-art methods. Beyond advancing fundamental visual reasoning, ShapeMoE provides a robust and efficient visual prior for downstream multimedia applications, particularly in occlusion-heavy immersive environments.
Figures & tables
Fig. 1: Motivation and Comparison of Routing Strategies. (a) One-size-fits-all models treat all shape types equally, often producing incomplete predictions under occlusion. (b) Naive MoE approaches rely on softmax-based routing without modeling shape distributions, leading to a mismatch between samples and experts. (c) Our ShapeMoE framework encodes each shape as a Gaussian distribution in a latent space, enabling shape-aware sparse routing to specialized experts and improving segmentation of diverse amodal shapes. Best viewed in color.
Fig. 2: Given an input image and a visible mask, the default single-Gaussian ShapeMoE performs amodal segmentation through the following stages. (1) The image is encoded by the image feature encoder, while the visible mask is embedded into a shape-aware mask embedding. (2) The Shape Distribution Encoder predicts the Gaussian parameters that characterize the object’s shape distribution in a learned latent space. (3) A latent shape representation is sampled, and the Shape-Aware Sparse Router computes expert selection scores to route each instance to the most appropriate expert. (4) The selected expert, specialized in specific shape patterns, predicts the final high-quality amodal segmentation mask. The GMM extension is described in Section III-G . Best viewed in color.
Fig. 3: Architecture of the Shape Distribution Encoder.
TABLE I: Comparison on amodal segmentation performance on three datasets, including COCOA-cls, D2SA, and KINS. SG and GMM denote the single-Gaussian baseline and its Gaussian mixture model extension, respectively. † denotes a zero-shot method which was pretrained on a large-scale amodal segmentation dataset [ 13 ] . Larger metric values of mIoU full and mIoU occ indicate better performance. Best performance is in bold, and second-best performance is underlined for each metric. A dash indicates an unreported result
Fig. 4: Qualitative results of the proposed ShapeMoE. Four representative cases are shown across various object categories, including bench, human, and horse, demonstrating ShapeMoE’s ability to handle complex occlusions and varied amodal shapes. Best viewed in color and zoomed in for details.
School of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China · School of Information Science and Engineering, Northeastern University, Shenyang 110819, China
College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China +2