cs.LGMar 13, 2026

Maximizing Incremental Information Entropy for Contrastive Learning

Authors: Jiansong Zhang, Zhuoqin Yang, Xu Wu, Xiaoling Luo, Peizhong Liu, Linlin Shen

Organizations: School of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · Computer Vision Institute, School of Artificial Intelligence, Shenzhen University, Shenzhen, China · School of Engineering, Huaqiao University, Quanzhou, China · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Contrastive learning has achieved remarkable success in self-supervised representation learning, often guided by information-theoretic objectives such as mutual information maximization. Motivated by the limitations of static augmentations and rigid invariance constraints, we propose IE-CL (Incremental-Entropy Contrastive Learning), a framework that explicitly optimizes the entropy gain between augmented views while preserving semantic consistency. Our theoretical framework reframes the challenge by identifying the encoder as an information bottleneck and proposes a joint optimization of two components: a learnable transformation for entropy generation and an encoder regularizer for its preservation. Experiments on CIFAR-10/100, STL-10, and ImageNet demonstrate that IE-CL consistently improves performance under small-batch settings. Moreover, our core modules can be seamlessly integrated into existing frameworks. This work bridges theoretical principles and practice, offering a new perspective in contrastive learning.

Figures & tables

Figure 1: Overview of the proposed IE-CL. We define incremental entropy as the absolute change in entropy induced by classical contrastive augmentations (see Definition 3.2 ). To optimize the contrastive learning process, we propose the S ample A ugmentation I ncremental B lock ( SAIB ), a learnable module that ensures the local Jacobian determinant > 1. By incorporating sample-level incremental entropy into contrastive optimization, we establish a principled framework that improves the effectiveness of self-supervised representation learning.
Method100 ep200 ep400 ep800 epBatch Size
SimCLR (ICML’20) Chen et al. (2020a)66.568.369.871.14096
SwAV (NeurIPS’20) Caron et al. (2020)66.569.170.771.04096
MoCo-v2 (CVPR’20) Chen et al. (2020c)67.469.970.971.3256
SimSiam (ICCV’21) Chen and He (2021)68.170.070.871.7256
NNCLR (ICCV’21) Dwibedi et al. (2021)65.466.166.868.71024
All4One (ICCV’23) Estepa et al. (2023)65.466.066.668.91024
Table 1: The comparison of the proposed method with ResNet-50 as the backbone under different numbers of pre-training iterations. Using BOLD and Underline formatting to highlight the best and second results.
Table 2: Comparison of self-supervised learning methods on various datasets (left) and segmentation/detection performance on PASCAL VOC2012 (right).
Figure 4
ConfigurationSAIBKL Reg.Encoder Reg.Top-1
MoCo-v2 (Baseline)✗✗✗52.50
+ Entropy Generation✓✗✗58.80
+ Semantic Consistency✓✓✗59.15
IE-CL (Full Framework)✓✓✓59.41
Table 3: Ablation study of the IE-CL components on ImageNet-1k using MoCo-v2 with a ResNet-18 backbone. We incrementally add our proposed components: the SAIB module for entropy generation, KL regularization for semantic consistency, and an Encoder Regularizer (implemented via Spectral Normalization) for entropy preservation.
Table 6
Figure 7
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Structure of Sample Augmentation Incremental Block ( SAIB ). Note that due to the patching of the 3-channel image, the batch occupies the position of the original channel. Therefore, it is possible to drive the inter-batch information to communicate by changing the original convolutional channels. All the convolutions in this block are cascaded with BN layers and Swish Ramachandran et al. (2017) to achieve nonlinear augmentation capability. And then convolved and non-linearly processed, and finally reconstructed back to the original position through positional coding.
Figure 7: Comparison of the time taken by different methods to train an epoch on ImageNet-1k with batch of 256. The proposed IE-CL, although it includes an additional non-isometric transform module SAIB, still spends less training cost compared to the previous methods because it uses momentum to update the Query encoder.
Figure 8: As a plug-and-play module, SAIB enhances the performance of existing contrastive learning methods with limited additional computational overhead. Overall, it achieves effective performance gains within an acceptable increase in training time—on average, approximately 8.8% more—compared to the original models (see Table 5 in the main text).
More in this topic

Contrastive Learning

Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.

See all 322 papers

Explore similar work

CardsList

  1. Positive Pair Geometry Matters: Optimal Transport for Contrastive Learning of Visual Representations

    Sep 21, 2026Akshit Nanda, Shahzad Ahmad, Ram Prasad PadhyContrastive LearningData Augmentation

    Department of Computer Science and Engineering, Indian Institute of Technology Bhubaneswar · Østfold University of Applied Sciences, NTNU

  2. Bayesian Gated Non-Negative Contrastive Learning

    May 27, 2026Peng Cui, Jiahao Zhang, Lijie HuContrastive LearningDisentangled Representation Learning

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI).

  3. InfoNCE Induces Gaussian Distribution

    Feb 27, 2026Roy Betser, Eyal Gofer, Meir Yossef Levi +1Contrastive LearningRepresentation Learning

    Technion - Israel Institute of Technology