With the widespread adoption of autonomous vehicles and robotics, amodal completion, which reconstructs the occluded parts of people and objects in an image, has become increasingly crucial. Just as humans infer hidden regions based on prior experience and common sense, this task inherently requires physical knowledge about real-world entities. However, existing approaches either depend solely on the image generation ability of visual generative models, which lack such knowledge, or leverage it only during the segmentation stage, preventing it from explicitly guiding the completion process. To address this, we propose AmodalCG, a novel framework that harnesses the real-world knowledge of Multimodal Large Language Models (MLLMs) to guide amodal completion. Our framework first assesses the extent of occlusion to selectively invoke MLLM guidance only when the target object is heavily occluded. If guidance is required, the framework further incorporates MLLMs to reason about both the (1) extent and (2) content of the missing regions. Finally, a visual generative model integrates these guidance and iteratively refines imperfect completions that may arise from inaccurate MLLM guidance. Experimental results on various real-world images show impressive improvements compared to all existing works, suggesting MLLMs as a promising direction for addressing challenging amodal completion.
Figures & tables
Figure 2: Completion results when meaningful parts of the target object are occluded. Stable Diffusion (SD) inpainting [ 17 ] often generates objects other than the target object. Existing amodal completion methods [ 15 , 21 , 1 ] lack an understanding of what should be generated for the missing parts. In contrast, our method provides explicit guidance on what should be reconstructed.
Figure 3: Overview of AmodalCG. Our framework first determines which samples would benefit from MLLM guidance ( Guidance Decision Module ). For those requiring guidance, the MLLM generates two key types of information about the occluded part of the target object: (1) the bounding box size of the full target object ( Geometric Guidance Module ) and (2) textual descriptions of the occluded region ( Semantic Guidance Module ). These are then incorporated into the completion process through a multi-scale expansion strategy, which selects the appropriate bounding box scale among the MLLM’s predictions ( Inpainting Module ).
Figure 4: Amodal completion results based on inpainting mask size. Unwanted objects are generated when the inpainting mask is substantially larger than the actual occluded region.
Figure 5: Example of the Geometric Guidance Module predicting multi-scale bounding boxes for the full target object.
Figure 6: Without masking occluders, MLLMs tend to describe occluders as well, resulting in extraneous generation in the final output. We highlight the semantic category of the target object in blue and the descriptions of the occluders in red .
Figure 7: Example of the Semantic Guidance Module generating detailed descriptions of occluded regions.
Figure 8: Qualitative evaluation of our method.
Method
COCO-A
BSDS-A
MP3D-A
Hard
Moderate
Easy
Hard
Moderate
Easy
Hard
Moderate
Easy
pix2gestalt
64.77
80.32
85.93
55.74
86.53
87.35
48.81
70.92
78.74
MC Diffusion
59.01
73.17
86.90
57.37
64.73
74.59
42.66
64.29
74.67
OWAAC
51.06
62.41
78.72
54.10
64.86
71.56
46.76
66.52
75.75
Ours
75.09
86.49
92.37
67.09
86.60
90.25
51.73
75.83
84.09
Table 1: Results on amodal segmentation by occlusion ratio.
Method
Occluded
Separated
Top 1 ↑
Top 3 ↑
Top 1 ↑
Top 3 ↑
No completion
34.00
49.26
21.10
34.70
pix2gestalt
43.39
58.97
31.15
45.77
MC Diffusion
44.74
62.07
34.50
49.72
OWAAC
40.50
55.30
27.83
40.97
Ours
45.06
62.99
40.01
56.70
Table 2: Qualitative evaluation on occluded object recognition using two datasets. We report Top 1 and Top 3 accuracy (%) using CLIP as a classification model.
COCO-A
BSDS-A
MP3D-A
GCR ↑
GSR ↑
GCR ↑
GSR ↑
GCR ↑
GSR ↑
94.18
48.73
98.17
35.02
99.62
-
Table 3: Guidance Call/Skip Rate (GCR/GSR) (%) of the Guidance Decision Module. GSR is not reported for MP3D-A, as it does not contain samples with low occlusion ( i.e . , < 10%).
Figure 9: Failure cases where the Guidance Decision Module failed to skip guidance.
Figure 10: Completion results across different mask scales.
Method
COCO-A
mIoU ↑ (%)
SD Inpainting (w/o guidance)
74.77
+ geometric guidance (single scale)
85.64
+ semantic guidance
85.60
+ multi-scale expansion (Ours)
86.31
Table 4: Ablation study of our framework. Each component is sequentially added.
Method
Occluded
Separated
Top 1 ↑
Top 3 ↑
Top 1 ↑
Top 3 ↑
Ours w/o semantic guidance
44.54
62.65
39.04
56.33
Ours
45.06
62.99
40.01
56.70
Table 5: Effectiveness of our semantic guidance in OOR.
Figure 11: (a) Image of the target object. (b) Completion results with a semantic category as a text prompt. (c) Completion results with a description generated without masking occluders. (d) Ours.
Figure 12: Occlusion percentage of the three datasets.
Figure 13: Examples of the three datasets.
Figure 14: A comparison of the descriptions generated by the MLLM when using visual prompts with and without occluder masking. In the descriptions, the semantic category of the target object is highlighted in bold , and descriptions of the occluders are highlighted in red .
Figure 15: Examples of inpainting masks generated by our method. The target object is highlighted with a red box in each image. The target objects are a house in the first image, a man in the second, and a car in the third.
Figure 16: MLLM occasionally includes descriptions of objects that are not present in the image but are related to the image context.
Method
COCO-A
BSDS-A
mIoU ↑ (%)
mIoU ↑ (%)
Original inpainting mask
48.00
52.02
Our inpainting mask
78.97
74.60
Table 6: Accuracy of our inpainting mask.
Figure 17: Examples of failure cases. Our method tends to avoid overly aggressive estimation, even in the case of coarse estimation. (a) Fails to predict the size of a long fence. (b) Matches the ground truth size well but still fails to predict a long fence.
Figure 18: Additional qualitative results of our method in amodal completion.
Figure 19: Additional qualitative results of our method in amodal completion.
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.