cs.CVAug 3, 2025

Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions

Authors: Zhixuan Li, Yujia Liu, Chen Hui, Jeonghaeng Lee, Sanghoon Lee, Weisi Lin

Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China · National Engineering Research Center of Visual Technology, Peking University, Beijing, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Amodal segmentation aims to predict complete object masks covering both visible and occluded regions, serving as a critical foundation for robust multimedia scene understanding and interactive media editing. This task poses significant challenges in complex visual media due to severe occlusions and extreme shape variation, ranging from rigid furniture to highly deformable clothing. Existing one-size-fits-all approaches struggle to reason about diverse amodal shapes in multimedia content due to limited representation capacity. While a mixture-of-experts framework is a natural solution, naively applying it without considering the underlying shape distribution leads to mismatched routing and redundant experts. To address these issues, we introduce ShapeMoE, a shape-specific sparse mixture-of-experts framework designed for complex visual scene parsing. The key idea is to learn a latent shape distribution space and dynamically route each object to a lightweight expert tailored to its geometric characteristics. Specifically, ShapeMoE encodes each object into a compact Gaussian embedding that captures key shape features. A Shape-Aware Sparse Router then maps the object to the most suitable expert, enabling precise shape-aware routing. Each expert is lightweight and specializes in recovering occluded regions for specific shape patterns. ShapeMoE is designed for shape-specific expert specialization while maintaining computational efficiency. Extensive experiments on three challenging benchmarks demonstrate that ShapeMoE consistently outperforms state-of-the-art methods. Beyond advancing fundamental visual reasoning, ShapeMoE provides a robust and efficient visual prior for downstream multimedia applications, particularly in occlusion-heavy immersive environments.

Figures & tables

Fig. 1: Motivation and Comparison of Routing Strategies. (a) One-size-fits-all models treat all shape types equally, often producing incomplete predictions under occlusion. (b) Naive MoE approaches rely on softmax-based routing without modeling shape distributions, leading to a mismatch between samples and experts. (c) Our ShapeMoE framework encodes each shape as a Gaussian distribution in a latent space, enabling shape-aware sparse routing to specialized experts and improving segmentation of diverse amodal shapes. Best viewed in color.
Fig. 2: Given an input image and a visible mask, the default single-Gaussian ShapeMoE performs amodal segmentation through the following stages. (1) The image is encoded by the image feature encoder, while the visible mask is embedded into a shape-aware mask embedding. (2) The Shape Distribution Encoder predicts the Gaussian parameters that characterize the object’s shape distribution in a learned latent space. (3) A latent shape representation is sampled, and the Shape-Aware Sparse Router computes expert selection scores to route each instance to the most appropriate expert. (4) The selected expert, specialized in specific shape patterns, predicts the final high-quality amodal segmentation mask. The GMM extension is described in Section III-G . Best viewed in color.
Fig. 3: Architecture of the Shape Distribution Encoder.
TABLE I: Comparison on amodal segmentation performance on three datasets, including COCOA-cls, D2SA, and KINS. SG and GMM denote the single-Gaussian baseline and its Gaussian mixture model extension, respectively. †{\dagger} denotes a zero-shot method which was pretrained on a large-scale amodal segmentation dataset [ 13 ] . Larger metric values of mIoU full{}_{\text{full}} and mIoU occ{}_{\text{occ}} indicate better performance. Best performance is in bold, and second-best performance is underlined for each metric. A dash indicates an unreported result
Fig. 4: Qualitative results of the proposed ShapeMoE. Four representative cases are shown across various object categories, including bench, human, and horse, demonstrating ShapeMoE’s ability to handle complex occlusions and varied amodal shapes. Best viewed in color and zoomed in for details.
IndexExpert Balancing LossCOCOA-clsD2SAKINS
1 87.6191.5588.06
2✓89.5392.4089.82
TABLE IV: Expert balancing loss in ShapeMoE (SG)
Table 7
More in this topic

Amodal Segmentation

See all 11 papers

Explore similar work

CardsList
May 23, 2026

Learnable Shape Prototypes with Occlusion-Geometry-Guided Injection for Amodal Instance Segmentation

Amodal instance segmentation aims to predict the complete object mask including occluded regions that lack pixel-level observations and must be inferred with the aid of shape priors. Existing methods acquire shape priors through fixed-capacity encoding spaces or expensive generative models, and inject them uniformly across all spatial positions without adapting to the varying prior demand between visible and occluded regions. In this paper, we propose a gated reliability-adaptive shape prior framework, which introduces a shape prior memory module that combines learnable prototypes via cross-attention to produce instance-adaptive shape priors through weighted prototype combination rather than generation. A spatial adaptive reliability gate then employs the signed distance field of the visible mask to modulate injection intensity at each position according to its occlusion depth, preserving reliable features in visible regions while directing shape compensation toward occluded areas. Experiments on two mainstream amodal instance segmentation benchmarks demonstrate that the proposed method outperforms existing approaches under multiple evaluation settings, improving the mean intersection-over-union over occluded regions by over 11 percentage points on one of the two benchmarks under the standard setting, while using approximately one-third of the total parameters. Linear probing analysis further reveals that the visible-mask cross-attention module implicitly encodes occlusion geometry into visual token representations, explaining the effectiveness of the proposed module decomposition.

Fufan Zhang, Jingxiang Wang, Xiangjie Ye

School of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China · School of Information Science and Engineering, Northeastern University, Shenyang 110819, China

Representation LearningOcclusion ReasoningInstance SegmentationPrototype LearningAmodal Segmentation
AskPDF
Apr 22, 2026

Amodal SAM: A Unified Amodal Segmentation Framework with Generalization

Amodal segmentation is a challenging task that aims to predict the complete geometric shape of objects, including their occluded regions. Although existing methods primarily focus on amodal segmentation within the training domain, these approaches often lack the generalization capacity to extend effectively to novel object categories and unseen contexts. This paper introduces Amodal SAM, a unified framework that leverages SAM (Segment Anything Model) for both amodal image and amodal video segmentation. Amodal SAM preserves the powerful generalization ability of SAM while extending its inherent capabilities to the amodal segmentation task. The improvements lie in three aspects: (1) a lightweight Spatial Completion Adapter that enables occluded region reconstruction, (2) a Target-Aware Occlusion Synthesis (TAOS) pipeline that addresses the scarcity of amodal annotations by generating diverse synthetic training data, and (3) novel learning objectives that enforce regional consistency and topological regularization. Extensive experiments demonstrate that Amodal SAM achieves state-of-the-art performance on standard benchmarks, while simultaneously exhibiting robust generalization to novel scenarios. We anticipate that this research will advance the field toward practical amodal segmentation systems capable of operating effectively in unconstrained real-world environments.

Bo Zhang, Zhuotao Tian, Xin Tao +3

Department of Computer Science, Harbin Institute of Technology at Shenzhen, Shenzhen 518057, China · Kuaishou Technology, Shenzhen, Guangdong, China

Vision Foundation Model AdaptationAmodal Segmentation
AskPDF
Aug 3, 2025

Single Point, Full Mask: Velocity-Guided Level Set Evolution for End-to-End Amodal Segmentation

Amodal segmentation aims to recover complete object shapes, including occluded regions, serving as an essential technique for user-centric multimedia authoring and object-level visual manipulation. Existing methods typically rely on informative prompts, such as bounding boxes or dense visible masks, which heavily degrade the user experience and interaction efficiency in real-world multimedia applications. While recent interactive paradigms (e.g., the Segment Anything Model) support lightweight point-based interactions, they often perform direct mask regression. Crucially, the opaque nature of these direct-regression models offers no visual explainability regarding how occluded structures are inferred, conflicting with the growing demand for interpretable multimedia systems. To address these limitations, we propose VELA, an end-to-end VElocity-driven Level-set Amodal segmentation method that enables explicit and transparent contour evolution driven by simple point clicks. VELA constructs an initial level set function from visual features and the user's point input, which then progressively evolves into the final amodal mask under the guidance of a shape-specific motion field predicted by a fully differentiable network. This mechanism learns to generate evolution dynamics at each step, ensuring that the spatial reasoning process is geometrically grounded, topologically flexible, and visually explainable to the user. Extensive experiments on COCOA-cls, D2SA, and KINS benchmarks demonstrate that VELA outperforms existing methods that use bounding-box or dense visible-mask prompts while requiring only a single-point prompt, validating the effectiveness of explainable geometric modeling for interactive multimedia tasks.

Zhixuan Li, Yujia Liu, Chen Hui +2

College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China +2

Promptable Image SegmentationAmodal Segmentation
AskPDF