Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

Jinen Setpal; Chaoyue Liu

cs.CVNov 5, 2026

Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

Authors: Jinen Setpal, Chaoyue Liu

Organizations: Elmore Family School of Electrical and Computer Engineering, Purdue University

Abstract

Image classification models often encounter objects in diverse and unpredictable contexts at test time, which may differ substantially from those seen during training. This phenomenon, termed context shift, exposes a key fragility in neural networks: when models rely on non-causal contextual cues for prediction, shifts in context can lead to significant performance degradation. In this paper, we propose a novel training framework to improve the robustness of convolutional neural networks under context shift. Our approach leverages spatial feature attribution to guide models toward making predictions that rely less on contextual regions and more on object-relevant features. As a first step, recognizing a theoretical limitation of existing feature attribution methods we introduce a new variant, ContrastiveCAMs, which produces more faithful attribution maps of model predictions. Building on ContrastiveCAMs, we further propose Context-Regularized Cross-Entropy (CR-CE), a modification of the standard cross-entropy loss that regularizes the model's attention by suppressing the influence of contextual regions, thereby improving robustness to context shift. We evaluate the effectiveness of our approach on several medium-to-large scale datasets (Waterbirds, Spawrious, ImageNet/ImageNet-BG, Hard-ImageNet), and report consistent improvements in context-shift robustness.

Figures & tables

Figure 1: Context Contributions Can Mislead Predictions. Spatial feature attribution maps reveal that cross-entropy trained models rely heavily on context, causing misclassifications in novel environments. Our approach encourages focus on the object thereby improving context-shift robustness.

Figure 2: Illustration of Theorem 3.2 . Shifting

\mathcal{A}

by arbitrary matrix

M

results in new explanations

\overline{\mathcal{A}}

. Although the explanation changed significantly, it corresponds to an identical final prediction

\tilde{f}

Figure 3: Qualitative Evaluation. Models trained using cross-entropy rely heavily on context: in the Hard-ImageNet (left) and ImageNet-BG (right) examples shown above, the baseball player and bee are predicted using the playing field ( $61.4\%$ of the prediction) and the flower ( $55.4\%$ of the prediction) respectively. On the other hand, models trained using CR-CE leverage object-specific regions and localize the target without the use of object-context segmentation during inference.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

CAM Size	Approach	No Shift Accuracy $(\uparrow)$	Accuracy under Target Removal			Rel. Foreground Sensitivity
CAM Size	Approach	No Shift Accuracy $(\uparrow)$	Gray Mask $(\downarrow)$	Gray BBOX $(\downarrow)$	Tile $(\downarrow)$	$\rho=0.25~(\uparrow)$	$\rho=0.5~(\uparrow)$
$7\times 7$	CE (Baseline)	$95.82$ $\pm 0.08$	$82.4$ $\pm 0.13$	$79.38$ $\pm 0.2$	$77.11$ $\pm 0.08$	$-0.21$ $\pm 0.01$	$-0.36$ $\pm 0.01$
	GALS	$95.91$ $\pm 0.15$	$80.93$ $\pm 0.27$	$76.62$ $\pm 1.21$	$75.69$ $\pm 0.73$	$-0.18$ $\pm 0.01$	$-0.31$ $\pm 0.03$
	EBL	$95.87$ $\pm 0.23$	$77.64$ $\pm 1.66$	$73.20$ $\pm 2.07$	$70.74$ $\pm 1.67$	$-0.11$ $\pm 0.04$	$-0.19$ $\pm 0.06$
	CFA	$\underline{96.00}$ $\pm 0.13$	$82.13$ $\pm 0.35$	$79.24$ $\pm 0.50$	$76.67$ $\pm 0.35$	$-0.20$ $\pm 0.01$	$-0.35$ $\pm 0.01$
	CR-CE (Ours)	$91.42$ $\pm 0.28$	$\mathbf{50.09}$ $\pm 0.89$	$\mathbf{45.78}$ $\pm 0.41$	$\mathbf{46.98}$ $\pm 0.2$	$\mathbf{+0.10}$ $\pm 0.02$	$\mathbf{+0.16}$ $\pm 0.01$

Appendix

Table 4: Hard-ImageNet Benchmark. Predictions on models trained with CR-CE are least likely to persist when the object is absent from the image. Object-focus increases proportional to the CAM resolution, with some performance cost incurred when the target is present alongside the context.

Figure 4: Superficially Aligned Attribution Maps. On a ResNet-50 trained with the GALS framework, we observe that HiResCAMs suggest a focus on the object regions; however, the model predicts the context-correlated class observed during training. ContrastiveCAMs faithfully reveal that the context contribution is in fact the primary driver of the prediction. This highlights the essential role that explanation faithfulness plays in ensuring improvements to context robustness in the explanation-guided learning paradigm.

Dataset	Epochs	Batch Size	LR	Momentum	Weight Decay	Scheduler
Waterbirds-100	$150$	$64$	$10^{-3}$	$0.9$	$10^{-4}$	-
Spawrious	$150$	$128$	$10^{-3}$		$10^{-4}$	-
Hard-ImageNet	$100$	$256$	$5\cdot 10^{-3}$		$10^{-4}$	-
ImageNet-1k / ImageNet-BG	$20$	$512$	$5\cdot 10^{-4}$		$10^{-5}$	Cosine Annealing

Appendix

Table 10: Hyperparameters are dataset-specific and kept consistent across approaches. Hard-ImageNet and ImageNet-1k are trained for fewer epochs since we fine-tune ImageNet-trained models. We increase the batch size as number of classes (2, 4, 15, and 1000 respectively) increases.

Figure 5: ImageNet-BG and Hard-ImageNet. Extended qualitative evaluations of object-context dependence by cross-entropy and CR-CE trained models. Images belonging to the Snorkel and Howler Monkey classes correspond to the Hard-ImageNet validation set, whereas the rest correspond to the ImageNet-BG test set. Across each of the examples, we observe the context dependence in models trained using cross-entropy loss is significantly mitigated by training with CR-CE.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 295 papers

ContextShift: A Controlled Benchmark for Context Dependence in Object Detection
Jun 8, 2026Dan Zlotnikov, Alex Lazarovich, Ohad Ben-ShaharObject Detection
Ben-Gurion University of the Negev, Israel
Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
Jul 7, 2026Suraj Yadav, Anjaneya Sharma, Siddharth YadavContrastive Learning Group Distributionally Robust Optimization
Indraprastha Institute of Information Technology Delhi
Evaluating Contextual Bias in CNN Image Classification: Evidence from Agricultural Benchmark Datasets
Sep 13, 2026Abhilekha Dalal, Michael Okonoda, Eder Martinez +1Image Classification Convolutional Neural Networks

Jun 8, 2026J/K move · Enter open · S save

ContextShift: A Controlled Benchmark for Context Dependence in Object Detection

Dan Zlotnikov, Alex Lazarovich, Ohad Ben-Shahar

Ben-Gurion University of the Negev, Israel

Object Detection

Open paper AskPDF arXiv

Modern object detectors achieve strong performance on standard benchmarks, yet their robustness to contextual variation remains insufficiently understood. Prior evaluations largely rely on aggregate metrics such as AP on uncontrolled distribution shifts, which can obscure how performance degrades under context change. We introduce ContextShift, a controlled benchmark that systematically manipulates object--context relationships while preserving object appearance. Built on COCO 2017, it isolates context as an independent variable through geometric transformations and synthetic and natural background substitutions, including a continuous compatibility axis based on normalized pointwise mutual information (NPMI). Across diverse detector architectures, we observe a consistent degradation pattern: false negatives increase by up to 227% and prediction volume decreases by up to 44%, while false positives remain stable or decline. This suppression behavior is not captured by aggregate metrics such as AP, which can mask substantial recall loss and changes in prediction dynamics. Further analysis suggests that degradation is driven less by reduced confidence than by a reduced formation of valid detection candidates. Moreover, performance along the statistical compatibility axis is non-monotonic, peaking at intermediate NPMI and degrading toward both extremes, indicating that statistical co-occurrence does not correlate linearly with effective visual context. Finally, we show that context-aware augmentation improves robustness: every augmented variant outperforms the dataset-only baseline on both original and manipulated test images, partially recovering performance lost to prediction-suppression failures by exposing models to object--context decoupling during training.

Abstract

Distribution Shift Robustness

Explore similar work

ContextShift: A Controlled Benchmark for Context Dependence in Object Detection

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Evaluating Contextual Bias in CNN Image Classification: Evidence from Agricultural Benchmark Datasets