Batch Optimization

Batch optimization techniques aim to accelerate the training of large neural networks by processing data in larger batches, thereby leveraging parallel computing resources. Current research focuses on adapting these techniques to various model architectures and tasks, including contrastive learning, reinforcement learning, and dense visual prediction, often employing algorithms like LARS, LAMB, and novel approaches like AGVM to mitigate challenges like gradient variance misalignment. These advancements significantly reduce training time for complex models, impacting fields like computer vision and natural language processing by enabling faster development and deployment of high-performing AI systems.

Papers

July 12, 2023

Mini-Batch Optimization of Contrastive Loss
Jaewoong Cho, Kartik Sreenivasan, Keon Lee, Kyunghoo Mun, Soheun Yi, Jeong-Gwan Lee, Anna Lee, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee
Contrastive Learning Loss Function Contrastive Loss Mini Batch Batch Optimization

November 20, 2022

Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, Sergey Kolesnikov
Neural Network Visual Analogue Scale Diverse Ensemble Batch Size Q Ensemble Batch Optimization

October 20, 2022

Large-batch Optimization for Dense Visual Predictions
Zeyue Xue, Jianming Liang, Guanglu Song, Zhuofan Zong, Liang Chen, Yu Liu, Ping Luo
Large Scale Large Batch Faster R CNN Dense Visual Prediction Adaptive Variance Batch Optimization

September 22, 2022

Batch Bayesian optimisation via density-ratio estimation with guarantees
Rafael Oliveira, Louis Tiao, Fabio Ramos
Bayesian Optimisation Formal Guarantee Approximate Bayesian Inference Probabilistic Classifier Density Ratio Estimation Batch Optimization

January 29, 2022

Batch Optimization

Papers

Mini-Batch Optimization of Contrastive Loss

Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size

Large-batch Optimization for Dense Visual Predictions

Batch Bayesian optimisation via density-ratio estimation with guarantees

Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning

ScaLA: Accelerating Adaptation of Pre-Trained Transformer-Based Language Models via Efficient Large-Batch Adversarial Noise