Organizations: DAVID Lab, UVSQ, Paris-Saclay University, Versailles, France · LIPN, UMR CNRS 7030, Sorbonne Paris Nord University, Villetaneuse, France · LISN, Paris-Saclay University, Saclay, France
Deep convolutional neural networks achieve remarkable performance by exhaustively processing dense spatial feature maps, yet this brute-force strategy introduces significant computational redundancy and encourages reliance on spurious background correlations. As a result, modern vision models remain brittle and difficult to interpret. We propose Energy-Regularized Spatial Masking (ERSM), a novel framework that reformulates feature selection as a differentiable energy minimization problem. By embedding a lightweight Energy-Mask Layer inside standard convolutional backbones, each visual token is assigned a scalar energy composed of two competing forces: an intrinsic Unary importance cost and a Pairwise spatial coherence penalty. Unlike prior pruning methods that enforce rigid sparsity budgets or rely on heuristic importance scores, ERSM allows the network to autonomously discover an optimal information-density equilibrium tailored to each input. We validate ERSM on convolutional architectures and demonstrate that it produces emergent sparsity, improved robustness to structured occlusion, and highly interpretable spatial masks, while preserving classification accuracy. Furthermore, we show that the learned energy ranking significantly outperforms magnitude-based pruning in deletion-based robustness tests, revealing ERSM as an intrinsic denoising mechanism that isolates semantic object regions without pixel-level supervision.
Figures & tables
Figure 1: Energy-Based Information Preservation. ERSM adaptively retains semantically relevant features while suppressing background noise.
Figure 2: Overview of the Energy-Regularized Spatial Masking (ERSM) framework. (a) A frozen feature extractor encodes the input image. (b–c) The feature map is partitioned into latent tokens. (d) Unary and 8-neighbor pairwise interactions define a token-wise energy. (e) The resulting spatial energy map suppresses redundant background regions. (f) The refined features are passed to the prediction head for classification.
Model
Acc. (%) ↑
E[mi]
Baseline CNN
69.14 ± 0.22
–
DropBlock
69.53 ± 0.22
–
Spatial Dropout
68.88 ± 0.34
–
L0-Gating
69.86 ± 0.30
0.65
ERSM-Unary (ours)
69.76 ± 0.15
0.65
ERSM-Full (ours)
69.73 ± 0.36
0.64
Table 1: Classification performance on Food-101 (Mean ± Std over 5 seeds).
Figure 3: Accuracy under progressive patch removal on the experimental CNN. ERSM preserves more accuracy than random deletion, peaking after removing 20%–25% of patches, highlighting its implicit feature-selection ability.
Baseline
ERSM
Dataset
Backbone
Res
P
Acc
Loss
Acc
Loss
CUB-200
ResNet-50
224
1
60.20
0.343
60.59
0.263
ResNet-50
256
1
59.82
0.374
60.67
0.274
ResNet-50
256
2
59.82
0.374
60.15
0.238
ResNet-50
448
1
45.81
0.880
47.67
0.611
ResNet-50
448
2
45.81
0.880
47.38
0.581
Table 2: Baseline (frozen) vs. ERSM across resolutions (Res) and patch sizes (P). Configurations where P does not divide the feature map are omitted. Peak values shown.
Figure 4: Robustness to Feature Removal on ResNet-50 Res 256 Patch 1. Comparison of classification accuracy under progressive patch deletion using Energy-based ranking (red) versus Random selection (blue).
Figure 5: ERSM Improvements. Left: The original image with baseline prediction (Wrong). Right: The ERSM prediction (Correct), showing the learned mask fading out the misleading background.
Figure 6: Failure Mode. (a) The model correctly finds the bird but misclassifies it (fine-grained error). (b) The model fails to mask the background, leading to confusion.
Electrical and Computer Engineering department at Clemson University, Clemson, SC · School of Computing at Binghamton University, Binghamton, NY · Department of Mathematical and Statistical Sciences at Clemson University, Clemson, SC