Severe occlusions of objects pose a major challenge for computer vision. We show that two root causes are (1) the loss of visible information and (2) the distracting patterns caused by the occluders. Our approach addresses both causes at the same time. First, the distracting patterns are removed at test-time, via masking of the occluding patterns. This masking is independent of the type of occlusion, by handling the occlusion through the lens of visual anomalies w.r.t. the object of interest. Second, to deal with less visual details, we follow standard practice by masking random parts of the object during training, for various degrees of occlusions. We discover that (a) it is possible to estimate the degree of the occlusion (i.e. severity) at test-time, and (b) that a model optimized for a specific degree of occlusion also performs best on a similar degree during test-time. Combining these two insights brings us to a severity-informed classification model called OASIC: Occlusion Agnostic Severity Informed Classification. We estimate the severity of occlusion for a test image, mask the occluder, and select the model that is optimized for the degree of occlusion. This strategy performs better than any single model optimized for any smaller or broader range of occlusion severities. Experiments show that combining gray masking with adaptive model selection improves AUCocc by +18.5 over standard training on occluded images and +23.7 over finetuning on unoccluded images.
Figures & tables
Figure 1: Occlusion handling by OASIC. At test time, an occlusion map is inferred by scoring against the memory bank M . The segmented mask is turned into gray to suppress distraction, while the estimated occlusion severity informs the selection of the most suitable classification model f∗ from the pool F , to better handle reduced visual information.
Figure 2: Comparison of an occluded image and its masked versions at different thresholds τ . From left to right: the original occluded image, and masks applied with thresholds 0.3, 0.5, and 0.7.
Figure 3: Larger degrees of occlusions (severity) deteriorate the performance of finegrained classification, trained without occlusions. Textured occlusions (vegetation, rubble) are more problematic than dull occlusion (smoke) or gray occlusion.
Figure 4: Textured occluders draw away the attention from the object. The first column shows the original (unoccluded) images, and the second column displays their corresponding saliency maps. The occlusion mask applied to each row is shown in column 3 and remains the same across all occlusion types. Columns 4–6 present the occluded images: gray, vegetation and rubble. Each is overlaid with its respective attention map.
Method
Vegetation
Smoke
Rubble
mAUROC
mAP
mAUROC
mAP
mAUROC
mAP
20% Occluded
AnomalyDINO
95.64 ± 1.51
79.35 ± 7.46
94.46 ± 2.33
76.59 ± 8.09
94.40 ± 2.34
74.56 ± 8.54
OVSeg
80.38 ± 9.09
63.25 ± 15.26
49.75 ± 5.55
24.16 ± 8.38
51.24 ± 6.30
25.54 ± 9.05
40% Occluded
AnomalyDINO
93.02 ± 3.33
88.02 ± 5.76
91.76 ± 3.51
85.52 ± 6.24
90.31 ± 4.47
82.21 ± 7.63
Table 1: Across various types of occlusions, the occlusion-agnostic AnomalyDINO performs better than OVSeg when prompted with the occlusion type. Mean AUROC (mAUROC) and mean Average Precision (mAP) is reported across occlusion severities (in rows) for occlusion types (vegetation, smoke, rubble; in columns).
Figure 5: OASIC is much more robust to severe occlusions, improving 5x compared to standard training. Both the gray-masking of occlusion (to suppress distraction) and the severity-informed model selection (to handle limited visibility) contribute to OASIC’s performance.
Panoptic segmentation in complex scenes remains challenging because of occlusions, yet modern approaches often neglect occlusion modelling. In this paper, we propose Position Embedding Modulation with Occlusion Level Attention (PEMOLA), a novel occlusion-aware module that can be seamlessly integrated into transformer-based panoptic segmentation. To obtain occlusion cues, we train an occlusion classifier on the COCO-OLAC dataset. The classifier derives the occlusion-level attention, which serves as spatial guidance, while the occlusion labels are encoded into a learnable embedding to produce channel-wise weights. Through joint modulation, PEMOLA elegantly introduces the occlusion priors into the position embedding, thereby improving the occlusion modelling. We further annotate the Cityscapes dataset with occlusion levels, termed Cityscapes Occlusion Labels for All Computer Vision Tasks (Cityscapes-OLAC), following the same labelling protocol as COCO-OLAC, to evaluate the cross-dataset generalisation ability of PEMOLA. Extensive experiments on COCO-OLAC and Cityscapes-OLAC demonstrate that PEMOLA consistently improves panoptic segmentation quality while introducing minimal computational overhead. These results highlight the importance of occlusion modelling, where incorporating occlusion-level attention helps deliver robust panoptic segmentation under occlusion. Code and dataset are available at https://github.com/wenbo-wei/PEMOLA.
Wenbo Wei, Jun Wang, Shan Raza +1
Department of Computer Science, University of Warwick, Coventry, UK
How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP, which were optimized for global image classification rather than localized segmentation. We introduce OVRCOAT, a simple, modular framework that tackles both. First, a CLIP-conditioned objectness adjustment (COAT) updates background/foreground probabilities, preserving high-quality masks for out-of-vocabulary objects. Second, an open-vocabulary mask-to-text refinement (OVR) strengthens CLIP's region-level alignment to improve classification of both seen and unseen classes with markedly lower memory cost than prior fine-tuning schemes. The two components combine to jointly improve objectness estimation and mask recognition, yielding consistent panoptic gains. Despite its simplicity, OVRCOAT sets a new state of the art on ADE20K (+5.5% PQ) and delivers clear gains on Mapillary Vistas and Cityscapes (+7.1% and +3% PQ, respectively). The code is available at: https://github.com/nickormushev/OVRCOAT
Nikolay Kormushev, Josip Šarić, Matej Kristan
University of Ljubljana Faculty of Comp. and Inf. Science · ETH Zurich Dept. of Computer Science · University of Zagreb Faculty of Elec. Eng. and Computing