cs.CVNov 6, 2026

Impact of color in computer and human-based object recognition under occlusion

Authors: Courtney M. King, Elissa Aminoff, Daniel D. Leeds

Organizations: Fordham University, Dept of Computer Science · Fordham University, Dept of Psychology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to CV. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition, and most transformers, are negatively impacted by generated camouflaging occluders, with scene-based camouflage performing worse than object-based. Classical CNNs show a strong opposite trend, performing the worst on neutral-colored occlusions, suggesting that contextually-related colors, particularly scene-based, benefit object recognition rather than disrupting shape or boundary cues. Additionally, generated camouflaging occluders for training augmentation improves robustness to unseen camouflage for selected CNN model, and occlusion metrics suggest trends about learned mechanisms. Findings have interesting implications for alignment in human and computer vision, model vulnerabilities, and explainability.

Figures & tables

Figure 1: Select images from Scene (top) and Object (middle) Occluders Dataset, with black (bottom) for control.
CategoryModelsKey features
CNNsResnet [ 35 ]skip connections
 VGG [ 36 ] )increased depth
 Googlenet [ 37 ]multiple filter sizes
TransformersViT [ 23 ]attention/patching
 Swin [ 38 ]hierarchial transformer
 MaxViT [ 39 ]attention+convolution
Table 1: Models categorized by primary mechanism: CNNs use convolution, transformers attention, and transformer-inspired CNNs aim to replicate the benefits of attention.
Figure 2: Clustering method for extracting dominant object, scene, and image colors.
Figure 3: Example trial in human behavioral experiment: each trial begins with a fixation cross for 300ms. An occluded image is shown, followed by an unoccluded version, each for 150ms. Images are separated by 150ms masks. At the end of each trial, participants are prompted to confirm whether the object is the same or not for up to three seconds. A three-second break is given after each trial.
Figure 4: Same/different images for two-alternative forced choice experiments.
Figure 5: Classification Accuracy of models on each color-occlusion type, for three occlusion placements: horizontal, vertical and diagonal. 95% confidence intervals are calculated by bootstrapping.
 Recognition AccuracyReaction Time
Original93.02% +/-0.49%457ms +/-13.86ms
Black87.36% +/-0.66%485ms +/-13.94ms
Object87.10% +/-0.67%487ms +/-14.01ms
Scene85.49% +/-0.70%491ms +/-13.78ms
Table 2: Mean accuracy and reaction time (+/- std) for human recognition for each occluded condition, for n=165n=165 participants.
  Recognition AccuracyReaction Time
type 1type 2P-correctedP-corrected
originalblack2.013876e-120.000202
originalobject1.027518e-140.000236
originalscene3.048062e-180.000001
blackobject7.381240e-011.000000
blackscene3.321923e-021.000000
Table 3: Pair-wise comparisons of recognition accuracy and reaction time with Holm correction. Those under threshold p<0.05p<0.05 shown in bold.
MetricModelBlackObjectScene
 Res (final)50.4% +/-3.75%48.9% +/-03.9%50.0% +/-4.4%
AIORes (mid)24.8% +/-5.7%25.0% +/-7.4%25.2% +/- 7.5%
low , highRes (early)50.8% +/-4.8%51.8% +/-4.4%53.0% +/-3.9%
 ViT (final)65.2% +/-18.9%56.0% +/-19.7%57.9% +/20.5%
 ViT (mid)44.0% +/-11.1%42.7% +/-12.3%42.4% +/-10.5%
 ViT (early)49.8% +/-9.2%37.2% +/-8.5%38.2% +/-8.5%
Table 4: Activation-evaluation metrics (+/- std) for multiple models/layers on each occluded dataset.
Figure 6: Activation maps from tiger image produced by Grad-CAM ++( [ 48 ] ) for multiple layers of ResNet 101 and ViT b16 .
Train setVal setResNet Acc. high , lowViT Acc. high , low
OriginalOriginal{\color[rgb]{0,0,1}65.52\%}_{[64.5,66.4]}{\color[rgb]{1,0,0}42.48\%}_{[41.5,43.4]}
Black-cutoutOriginal66.56%[65.6,67.5]66.56\%_{[65.6,67.5]}{\color[rgb]{0,0,1}41.73\%}_{[40.7,42.6]}
Color-cutoutOriginal{\color[rgb]{1,0,0}66.85\%}_{[65.9,67.7]}41.76%[40.8,42.7]41.76\%_{[40.8,42.7]}
OriginalColor{\color[rgb]{0,0,1}61.57\%}_{[60.6,62.5]}38.95%[38,39.9]38.95\%_{[38,39.9]}
Black-cutoutColor62.30%[61.3,63.2]62.30\%_{[61.3,63.2]}{\color[rgb]{0,0,1}38.53\%}_{[37.5,39.4]}
Color-cutoutColor{\color[rgb]{1,0,0}65.7\%}_{[64.7,66.6]}{\color[rgb]{1,0,0}41.27\%}_{[40.3,42.2]}
Table 5: Augmentation performance on Tiny-ImageNet for ResNet 18 and ViT 16-tiny with Cutout in black and image-relevant colors. 95% confidence intervals calculated by bootstrapping.
Figure 7: Activation maps of the final layer of ResNet 18 models trained without augmentation, with cutout-black, and cutout-color, show on an image of each occlusion type.
MetricAugment.ResNet (final)ViT (final)
 None49.13% 6.15%42.43% 13.60%
AIOCutout-Black49.51% 6.07%42.64% 12.92%
low , highCutout-Color48.26% 4.15%39.65% 10.65%
 None58.03% 18.80%46.71% 11.04%
NOPCutout-Black59.45% 18.48%47.93% 9.48%
high , lowCutout-Color71.04% 13.31%47.14% 9.69%
Table 6: Activation-evaluation metrics (+/- std) on augmented models (rows 1-3 of Table 5 ) tested on Tiny-ImageNet validation set with image-relevant color occlusions.
Figure 8: Pixel sets (left) and metrics (right) as defined in Section 3.4 Equations 1 , 2 , 3 .
  Recog. Acc.  React. Time  
type 1type 2tPP-correctedtPP-corrected
originalblack7.8527165.034689e-132.013876e-12-4.2194974.042312e-050.000202
originalobject8.7807492.055035e-151.027518e-14-4.1237065.910685e-050.000236
originalscene10.1247425.080103e-193.048062e-18-5.4657791.686844e-070.000001
blackobject0.3349067.381240e-017.381240e-01-0.2242108.228730e-011.000000
blackscene2.5695601.107308e-023.321923e-02-0.8448983.993985e-011.000000
Table 7: Pair-wise comparisons of recognition accuracy and reaction time. Comparisons under the threshold p<0.05p<0.05 after Holm correction are shown in bold.
Figure 9: Activation maps from oven image of select models produced by Grad-CAM ++( [ 47 ] ). The unoccluded images were correctly classified by both models. Most occluded instances were misclassified, with the exception of the black occluded image by ViT.
 F ValuePr << Fηp2\eta^{2}_{p}
Recog. Acc.39.457.330964e-230.19392
React. time10.4639380.0000010.05998
Table 8: Significance of differences among four occlusion conditions with repeated-measures one-way ANOVA.
MetricAugment.ResNet (final)ViT (final)
 None50.04% +/-6.21%42.04% +/-14.71
AIOCutout-Black51.87% +/-6.93%45.24% +/-14.77%
 Cutout-Color48.35% +/-4.20%37.19% +/-9.84%
 None45.96% +/-20.04%40.38% +/-11.28%
NOPCutout-Black46.19% +/-19.81%41.54% +/-10.09%
 Cutout-Color62.16% +/-18.98%41.68% +/-10.46%
Table 9: Activation-evaluation metrics (+/- std) on augmented models (rows 1-3 of Table 5 ) tested on Tiny-ImageNet test set with image-relevant color k=5k=5 occlusions as horizontal stripes at α=0.5\alpha=0.5 .
Figure 10: Activation maps of an early layer of ResNet 18 models trained without augmentation, with cut-out black, and cut-out color, show on an image of each occlusion type.
More in this topic

Object Recognition

See all 16 papers

Explore similar work

CardsList
Apr 5, 2026

OASIC: Occlusion-Agnostic and Severity-Informed Classification

Severe occlusions of objects pose a major challenge for computer vision. We show that two root causes are (1) the loss of visible information and (2) the distracting patterns caused by the occluders. Our approach addresses both causes at the same time. First, the distracting patterns are removed at test-time, via masking of the occluding patterns. This masking is independent of the type of occlusion, by handling the occlusion through the lens of visual anomalies w.r.t. the object of interest. Second, to deal with less visual details, we follow standard practice by masking random parts of the object during training, for various degrees of occlusions. We discover that (a) it is possible to estimate the degree of the occlusion (i.e. severity) at test-time, and (b) that a model optimized for a specific degree of occlusion also performs best on a similar degree during test-time. Combining these two insights brings us to a severity-informed classification model called OASIC: Occlusion Agnostic Severity Informed Classification. We estimate the severity of occlusion for a test image, mask the occluder, and select the model that is optimized for the degree of occlusion. This strategy performs better than any single model optimized for any smaller or broader range of occlusion severities. Experiments show that combining gray masking with adaptive model selection improves AUCocc\text{AUC}_\text{occ} by +18.5 over standard training on occluded images and +23.7 over finetuning on unoccluded images.

Kay Gijzen, Gertjan J. Burghouts, Daniël M. Pelt

Leiden University · TNO

Image ClassificationOcclusion Reasoning
AskPDF
Aug 11, 2026

Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection

Camouflaged object detection (COD) aims to segment objects that are visually concealed in their surroundings and has attracted increasing attention in recent years. However, most existing COD methods are developed under a closed-world assumption, where each input image is assumed to contain a camouflaged object. This assumption ignores realistic scenarios with pure backgrounds or non-camouflaged objects, causing existing models to produce severe false positives when deployed in open-world environments. To address this limitation, we propose OPC16K, a large-scale benchmark for realistic COD. OPC16K contains 16,245 images from 14 sources and is carefully organized into camouflaged-object images, pure background images, and non-camouflaged-object images, enabling comprehensive evaluation of both segmentation quality and negative-sample rejection. Based on this benchmark, we further propose OPCNet, a presence-aware camouflage network that reformulates COD from a pure segmentation task into a joint problem of object localization and camouflage existence reasoning. Specifically, OPCNet introduces hierarchical existence reasoning to distinguish CO, BG, and NOCOD scenarios, similarity-aware camouflage relation modeling to capture foreground-background camouflage cues, and existence-aware feature refinement to regulate segmentation features with existence predictions. Extensive experiments on OPC16K demonstrate that OPCNet achieves superior performance under the proposed realistic COD evaluation protocol, significantly reducing false positives on negative samples while maintaining accurate camouflaged-object segmentation. Code and dataset will be released at https://github.com/2231122/OPCOD.

Huafeng Chen, Yueming Lyu, Chenyang Si +3

School of Intelligence Science and Technology, Nanjing University · Department of Computing, Imperial College London

Unsupervised Camouflaged Object DetectionObject DetectionCamouflaged Object Detection
AskPDF
May 15, 2026

SeamCam: Quantifying Seamless Camouflage via Multi-Cue Visual Detectability

Animals are described as effectively camouflaged when they blend seamlessly with their surrounding, yet no standardized quantitative measure of this seamlessness exists. We address this gap by framing camouflage evaluation as a visual localization problem: a well-camouflaged animal is one that remains difficult to detect even when its category is known. We introduce SeamCam (Seamless Camouflage), a metric that quantifies how detectable an animal is from the available visual evidence. Given an image and a target species, SeamCam generates category-conditioned detection proposals, extracts segmentation masks, and identifies the subset whose collective union yields the highest IoU with the ground-truth mask. The SeamCam score is one minus this maximum recoverable localization signal, where a higher score indicates stronger camouflage (i.e., lower detectability). In a human two-alternative forced-choice study with 94 participants and 2,390 comparisons, SeamCam achieves 78.82% agreement with human camouflage difficulty judgments, outperforming state-of-the-art by about 25%. We then demonstrate SeamCam's utility as a preference signal for Direct Preference Optimization (DPO) to fine-tune a diffusion-based inpainting model for camouflage generation. This offers an affordable training approach with an objective explicitly suited for camouflage generation, unlike typical diffusion models. To support rigorous benchmarking, we further introduce CamFG-1.5k, a curated dataset of 1,521 high-resolution images in which animals are fully visible prior to camouflage generation, enabling unbiased evaluation by controlling for occlusion artifacts present in existing datasets. https://7amin.github.io/SeamCam/

Amin Karimi Monsefi, Abolfazl Meyarian, Mridul Khurana +6

The Ohio State University · Path Robotics, USA · Virginia Tech +2

Camouflaged Object Detection
AskPDF