Towards Reconstructing and Recognizing Objects behind Fragmented Occlusion using Video
Authors: Roman Pflugfelder, Michael Lindenbaum, Daniel Cremers
Organizations: Austrian Institute of Technology Digital Safety & Security Department · Technical University of Munich School of Computation, Information and Technology Munich Center for Machine Learning (MCML) · Technion Faculty of Computer Science
Perceiving objects through fragmented occlusions, such as foliage, is a natural capability of the human visual system but remains a fundamental challenge for computer vision. This work addresses the problem of object reconstruction, or de-occlusion, in the presence of high-density, fragmented occlusions. We propose a reconstruction framework that operates on a sequence of grayscale images of a moving object, acquired by a single camera, either stationary or in motion. The algorithm estimates the object's relative positions across frames by jointly registering the images and optimizing the alignment parameters to maximize the contrast of their average, referred to as the integrated image. This integrated image serves as the reconstructed representation of the object. Experimental evaluations on both synthetic and real video sequences demonstrate the feasibility of accurate reconstruction under occlusion densities exceeding 90%. The proposed approach is competitive with synthetic aperture techniques, which are limited to static objects and require complex camera setup with careful calibration. Furthermore, we show that the integrated image enables reliable object classification under severe occlusion when used as input to standard, off-the-shelf classifiers.
Figures & tables
ρ
T
ν
1
11.67±0.50
25.24±12.32
100.0±0.00
2
28.91±1.01
25.40±12.26
100.0±0.00
3
51.20±1.11
23.36±09.12
100.0±0.00
4
70.22±1.59
24.44±10.95
99.83±2.42
5
82.57±1.14
23.98±10.66
96.64±2.61
6
90.51±1.87
25.91±12.13
84.00±8.69
Table 1: Synthetic video statistics: The image sequences are divided into six sets associated, each, with roughly the same occlusion density ρ ( mean±std. deviation of each quantity). The sequence length T and the object’s total visibility ν are also given.
Figure 3: Experimental results: (a) The plots draw μMSE (square), μP^ ( cross ), μP ( circle ) with confidence interval ±3σ (ordinate) vs . ρ (abscissa). (b) Again μMSE vs . ρ , but we provide information on the sequence length T : Dark blue encodes long sequences. Longer sequences do not necessarily leader to a better reconstruction.
Figure 4: Five synthetic sample videos: (a) quantitative data. (b) - (f) Raw occluded images (top row) and the object reconstructions for the different occlusions densities (bottom row).
Figure 5: Recognition with CLIP: (a) The distance D vs . the occlusion density ρ for the occluded images. The mean and confidence interval of ape is shown in blue. (b) The same evaluation for the reconstructed images, shows a significant improvement. (c) The CLIP’s classification error rate vs . ρ .
Figure 6: Park results: (a) Reference image and fixation window. (b) Applying CLIP to (a). The bar plot shows the four smallest cosine distances for label human figure and the same 24 diverse outdoor categories as used for the synthetic videos. (c) object reconstruction, (d) applying CLIP to (c). Note the correct ranking compared to (b). (e) fore- and background reconstruction. (f) - (h) integral images corresponding to some incorrect motions.
Figure 7: Both plots draw f (ordinate) vs . α (abscissa). (a) Two significant maxima of the person and static motions are observable. (b) The overlaps of fence slats cause misleading maxima.
Figure 8: Boreal forest and camera pan results: (a) and (c) ref. images and fixation windows. (b) and (d) object reconstruction. (e) foreground (tree) reconstruction related to camera panning.
Figure 9: Fence results: (a) Reference image and fixation window. (b) - (f) Integral images for different motions. (e) fore- and background reconstruction. (f) car reconstruction.
Figure 10: Park sequence histograms.
Figure 11: Fence sequence histograms.
Figure 12: Evaluation of different motions. Top: The variance of (a) and (c) is larger or equal to (b), although (b) shows higher contrast than (a) and (c) and is the integrated image of the correct box’s motion. Bottom: This is verified by the histogram spreads. (d) illustrates the static occluder. The histogram spread for (d) is maximal (255) due to the chosen brightness values for the background and occluder.
Figure 13: Box experiment: (a) Ground truth reference image. (b) Sequence’s reference image at reference r=11 . (c) 2D visualization of f - β (abscissa) vs . α (ordinate): Colors encode the variance. The motion at (αβ)⊤=0 is found by the algorithm ( + ). f attains no maximum for the box’s motion (−55)⊤ .
Figure 14: Evaluation of the fixation window size: (a) Windows of varying size. (b) Sample variance computed in the integrated image for the different windows. Note that the sample variance for the correct motion is significantly higher than for the incorrect motions when the fixation window closely catches the object.
Figure 15: Experiment with multiple objects: (a) - (e) For varying occlusion densities, the images show the ground truth Iˉr (top), raw data Ir (middle) and reconstructed images I^r (bottom). (f) Plot of PSNR (dB) vs . occlusion density. Colors are the same as in (g) Deviation (RMSE) between object and occluder intensities. Colors encode the six objects.
Recovering the complete appearance of occluded objects from a single image is a fundamental problem for scene understanding and image editing. Real-world scenes are challenging due to clutter, complex spatial occlusions, and diverse object categories. Existing methods often struggle when occlusion involves background surfaces or objects that are not explicitly detected, and fail to resolve the cyclic occlusion dependencies that frequently occur in natural scenes. We introduce a training-free, open-world scene deocclusion framework that recovers the full amodal appearance of detected or user-selected objects from a single RGB image, decomposing the scene into depth-ordered RGBA layers that are geometrically plausible, semantically faithful, and visually consistent with the visible scene. Extensive experiments show that our approach improves scene-level deocclusion, producing coherent and realistic reconstructions in real-world settings. Code will be available on the project page: https://sashjk.github.io/owsd/
K. M. S. J. Kumarasinghe, Neil A. Dodgson, Fang-Lue Zhang
Victoria University of Wellington · University of New South Wales
Severe occlusions of objects pose a major challenge for computer vision. We show that two root causes are (1) the loss of visible information and (2) the distracting patterns caused by the occluders. Our approach addresses both causes at the same time. First, the distracting patterns are removed at test-time, via masking of the occluding patterns. This masking is independent of the type of occlusion, by handling the occlusion through the lens of visual anomalies w.r.t. the object of interest. Second, to deal with less visual details, we follow standard practice by masking random parts of the object during training, for various degrees of occlusions. We discover that (a) it is possible to estimate the degree of the occlusion (i.e. severity) at test-time, and (b) that a model optimized for a specific degree of occlusion also performs best on a similar degree during test-time. Combining these two insights brings us to a severity-informed classification model called OASIC: Occlusion Agnostic Severity Informed Classification. We estimate the severity of occlusion for a test image, mask the occluder, and select the model that is optimized for the degree of occlusion. This strategy performs better than any single model optimized for any smaller or broader range of occlusion severities. Experiments show that combining gray masking with adaptive model selection improves AUCocc by +18.5 over standard training on occluded images and +23.7 over finetuning on unoccluded images.
Object-centric scene understanding is a fundamental challenge in computer vision. Existing approaches often rely on multi-stage pipelines that first apply pre-trained segmentors to extract individual objects, followed by per-object 3D reconstruction. Such methods are computationally expensive, fragile to segmentation errors, and scale poorly with scene complexity. We introduce OCH3R, a unified framework for Object-Centric Holistic 3D Reconstruction from a single RGB image. OCH3R performs one forward pass to simultaneously predict all object instances with their 6D poses and detailed 3D reconstructions. The key idea is a transformer architecture that predicts per-pixel attributes, including CLIP-based category embeddings, metric depth, normalized object coordinates (NOCS), and a fixed number of 3D Gaussians representing each object. To supervise these Gaussian reconstructions, we transform them into canonical space using the predicted 6D poses and align them with pre-rendered canonical ground truth, avoiding costly per-image Gaussian label generation. On standard indoor benchmarks, OCH3R achieves state-of-the-art performance across monocular depth estimation, open-vocabulary semantic segmentation, and RGB-only category-level 6D pose estimation, while producing high-fidelity, editable per-object reconstructions. Crucially, inference is fully feed-forward and scales independently of the number of objects, offering orders-of-magnitude speedups over conventional multi-stage pipelines in cluttered scenes.