Impact of color in computer and human-based object recognition under occlusion

Courtney M. King; Elissa Aminoff; Daniel D. Leeds

cs.CVNov 6, 2026

Impact of color in computer and human-based object recognition under occlusion

Authors: Courtney M. King, Elissa Aminoff, Daniel D. Leeds

Organizations: Fordham University, Dept of Computer Science · Fordham University, Dept of Psychology

Abstract

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to CV. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition, and most transformers, are negatively impacted by generated camouflaging occluders, with scene-based camouflage performing worse than object-based. Classical CNNs show a strong opposite trend, performing the worst on neutral-colored occlusions, suggesting that contextually-related colors, particularly scene-based, benefit object recognition rather than disrupting shape or boundary cues. Additionally, generated camouflaging occluders for training augmentation improves robustness to unseen camouflage for selected CNN model, and occlusion metrics suggest trends about learned mechanisms. Findings have interesting implications for alignment in human and computer vision, model vulnerabilities, and explainability.

Figures & tables

Panel of Figure 1 — Figure 1: Select images from Scene (top) and Object (middle) Occluders Dataset, with black (bottom) for control.

Category	Models	Key features
CNNs	Resnet [ 35 ]	skip connections
	VGG [ 36 ] )	increased depth
	Googlenet [ 37 ]	multiple filter sizes
Transformers	ViT [ 23 ]	attention/patching
	Swin [ 38 ]	hierarchial transformer
	MaxViT [ 39 ]	attention+convolution

Table 1: Models categorized by primary mechanism: CNNs use convolution, transformers attention, and transformer-inspired CNNs aim to replicate the benefits of attention.

Figure 2: Clustering method for extracting dominant object, scene, and image colors.

Figure 3: Example trial in human behavioral experiment: each trial begins with a fixation cross for 300ms. An occluded image is shown, followed by an unoccluded version, each for 150ms. Images are separated by 150ms masks. At the end of each trial, participants are prompted to confirm whether the object is the same or not for up to three seconds. A three-second break is given after each trial.

Figure 4: Same/different images for two-alternative forced choice experiments.

Panel of Figure 5 — Figure 5: Classification Accuracy of models on each color-occlusion type, for three occlusion placements: horizontal, vertical and diagonal. 95% confidence intervals are calculated by bootstrapping.

	Recognition Accuracy	Reaction Time
Original	93.02% +/-0.49%	457ms +/-13.86ms
Black	87.36% +/-0.66%	485ms +/-13.94ms
Object	87.10% +/-0.67%	487ms +/-14.01ms
Scene	85.49% +/-0.70%	491ms +/-13.78ms

Table 2: Mean accuracy and reaction time (+/- std) for human recognition for each occluded condition, for

n=165

participants.

		Recognition Accuracy	Reaction Time
type 1	type 2	P-corrected	P-corrected
original	black	2.013876e-12	0.000202
original	object	1.027518e-14	0.000236
original	scene	3.048062e-18	0.000001
black	object	7.381240e-01	1.000000
black	scene	3.321923e-02	1.000000

Table 3: Pair-wise comparisons of recognition accuracy and reaction time with Holm correction. Those under threshold

p<0.05

shown in bold.

Metric	Model	Black	Object	Scene
	Res (final)	50.4% +/-3.75%	48.9% +/-03.9%	50.0% +/-4.4%
AIO	Res (mid)	24.8% +/-5.7%	25.0% +/-7.4%	25.2% +/- 7.5%
low , high	Res (early)	50.8% +/-4.8%	51.8% +/-4.4%	53.0% +/-3.9%
	ViT (final)	65.2% +/-18.9%	56.0% +/-19.7%	57.9% +/20.5%
	ViT (mid)	44.0% +/-11.1%	42.7% +/-12.3%	42.4% +/-10.5%
	ViT (early)	49.8% +/-9.2%	37.2% +/-8.5%	38.2% +/-8.5%

Table 4: Activation-evaluation metrics (+/- std) for multiple models/layers on each occluded dataset.

Panel of Figure 6 — Figure 6: Activation maps from tiger image produced by Grad-CAM ++( [ 48 ] ) for multiple layers of ResNet 101 and ViT b16 .

Train set	Val set	ResNet Acc. high , low	ViT Acc. high , low
Original	Original	${\color[rgb]{0,0,1}65.52\%}_{[64.5,66.4]}$	${\color[rgb]{1,0,0}42.48\%}_{[41.5,43.4]}$
Black-cutout	Original	$66.56\%_{[65.6,67.5]}$	${\color[rgb]{0,0,1}41.73\%}_{[40.7,42.6]}$
Color-cutout	Original	${\color[rgb]{1,0,0}66.85\%}_{[65.9,67.7]}$	$41.76\%_{[40.8,42.7]}$
Original	Color	${\color[rgb]{0,0,1}61.57\%}_{[60.6,62.5]}$	$38.95\%_{[38,39.9]}$
Black-cutout	Color	$62.30\%_{[61.3,63.2]}$	${\color[rgb]{0,0,1}38.53\%}_{[37.5,39.4]}$
Color-cutout	Color	${\color[rgb]{1,0,0}65.7\%}_{[64.7,66.6]}$	${\color[rgb]{1,0,0}41.27\%}_{[40.3,42.2]}$

Table 5: Augmentation performance on Tiny-ImageNet for ResNet 18 and ViT 16-tiny with Cutout in black and image-relevant colors. 95% confidence intervals calculated by bootstrapping.

Panel of Figure 7 — Figure 7: Activation maps of the final layer of ResNet 18 models trained without augmentation, with cutout-black, and cutout-color, show on an image of each occlusion type.

Metric	Augment.	ResNet (final)	ViT (final)
	None	49.13% 6.15%	42.43% 13.60%
AIO	Cutout-Black	49.51% 6.07%	42.64% 12.92%
low , high	Cutout-Color	48.26% 4.15%	39.65% 10.65%
	None	58.03% 18.80%	46.71% 11.04%
NOP	Cutout-Black	59.45% 18.48%	47.93% 9.48%
high , low	Cutout-Color	71.04% 13.31%	47.14% 9.69%

Table 6: Activation-evaluation metrics (+/- std) on augmented models (rows 1-3 of Table 5 ) tested on Tiny-ImageNet validation set with image-relevant color occlusions.

Panel of Figure 8 — Figure 8: Pixel sets (left) and metrics (right) as defined in Section 3.4 Equations 1 , 2 , 3 .

		Recog. Acc.			React. Time
type 1	type 2	t	P	P-corrected	t	P	P-corrected
original	black	7.852716	5.034689e-13	2.013876e-12	-4.219497	4.042312e-05	0.000202
original	object	8.780749	2.055035e-15	1.027518e-14	-4.123706	5.910685e-05	0.000236
original	scene	10.124742	5.080103e-19	3.048062e-18	-5.465779	1.686844e-07	0.000001
black	object	0.334906	7.381240e-01	7.381240e-01	-0.224210	8.228730e-01	1.000000
black	scene	2.569560	1.107308e-02	3.321923e-02	-0.844898	3.993985e-01	1.000000

Table 7: Pair-wise comparisons of recognition accuracy and reaction time. Comparisons under the threshold

p<0.05

after Holm correction are shown in bold.

Panel of Figure 9 — Figure 9: Activation maps from oven image of select models produced by Grad-CAM ++( [ 47 ] ). The unoccluded images were correctly classified by both models. Most occluded instances were misclassified, with the exception of the black occluded image by ViT.

	F Value	Pr $<$ F	$\eta^{2}_{p}$
Recog. Acc.	39.45	7.330964e-23	0.19392
React. time	10.463938	0.000001	0.05998

Table 8: Significance of differences among four occlusion conditions with repeated-measures one-way ANOVA.

Metric	Augment.	ResNet (final)	ViT (final)
	None	50.04% +/-6.21%	42.04% +/-14.71
AIO	Cutout-Black	51.87% +/-6.93%	45.24% +/-14.77%
	Cutout-Color	48.35% +/-4.20%	37.19% +/-9.84%
	None	45.96% +/-20.04%	40.38% +/-11.28%
NOP	Cutout-Black	46.19% +/-19.81%	41.54% +/-10.09%
	Cutout-Color	62.16% +/-18.98%	41.68% +/-10.46%

Table 9: Activation-evaluation metrics (+/- std) on augmented models (rows 1-3 of Table 5 ) tested on Tiny-ImageNet test set with image-relevant color

k=5

occlusions as horizontal stripes at

\alpha=0.5

Panel of Figure 10 — Figure 10: Activation maps of an early layer of ResNet 18 models trained without augmentation, with cut-out black, and cut-out color, show on an image of each occlusion type.

Object Recognition

See all 16 papers

OASIC: Occlusion-Agnostic and Severity-Informed Classification
Apr 5, 2026Kay Gijzen, Gertjan J. Burghouts, Daniël M. PeltImage Classification Occlusion Reasoning
Leiden University · TNO
Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection
Aug 11, 2026Huafeng Chen, Yueming Lyu, Chenyang Si +3Unsupervised Camouflaged Object Detection Object Detection
School of Intelligence Science and Technology, Nanjing University · Department of Computing, Imperial College London
SeamCam: Quantifying Seamless Camouflage via Multi-Cue Visual Detectability
May 15, 2026Amin Karimi Monsefi, Abolfazl Meyarian, Mridul Khurana +6Camouflaged Object Detection
The Ohio State University · Path Robotics, USA · Virginia Tech +2

Apr 5, 2026J/K move · Enter open · S save

OASIC: Occlusion-Agnostic and Severity-Informed Classification

Kay Gijzen, Gertjan J. Burghouts, Daniël M. Pelt

Leiden University · TNO

Image Classification Occlusion Reasoning

Open paper AskPDF arXiv

Severe occlusions of objects pose a major challenge for computer vision. We show that two root causes are (1) the loss of visible information and (2) the distracting patterns caused by the occluders. Our approach addresses both causes at the same time. First, the distracting patterns are removed at test-time, via masking of the occluding patterns. This masking is independent of the type of occlusion, by handling the occlusion through the lens of visual anomalies w.r.t. the object of interest. Second, to deal with less visual details, we follow standard practice by masking random parts of the object during training, for various degrees of occlusions. We discover that (a) it is possible to estimate the degree of the occlusion (i.e. severity) at test-time, and (b) that a model optimized for a specific degree of occlusion also performs best on a similar degree during test-time. Combining these two insights brings us to a severity-informed classification model called OASIC: Occlusion Agnostic Severity Informed Classification. We estimate the severity of occlusion for a test image, mask the occluder, and select the model that is optimized for the degree of occlusion. This strategy performs better than any single model optimized for any smaller or broader range of occlusion severities. Experiments show that combining gray masking with adaptive model selection improves

\text{AUC}_\text{occ}

by +18.5 over standard training on occluded images and +23.7 over finetuning on unoccluded images.

Abstract

Object Recognition

Explore similar work

OASIC: Occlusion-Agnostic and Severity-Informed Classification

Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection

SeamCam: Quantifying Seamless Camouflage via Multi-Cue Visual Detectability