Independent Minibatching

Independent minibatching, a core technique in stochastic gradient descent (SGD), focuses on efficiently training machine learning models by processing subsets of data in parallel. Current research investigates optimizing minibatch construction, exploring methods like fixed-size sampling (with and without replacement) and determinantal point processes to reduce variance and improve convergence, particularly for second-order optimizers and in challenging scenarios such as differentially private training and graph neural networks. These advancements aim to enhance the speed, scalability, and privacy guarantees of training large models, impacting both the efficiency of machine learning algorithms and the feasibility of deploying them in resource-constrained environments.