Memory-Efficient Training

Memory-efficient training studies how to reduce the memory used by model weights, activations, gradients, and optimizer states so larger models or longer sequences can be trained on limited hardware. Common approaches include quantizing or factorizing optimizer states and activations, sparse updates, activation checkpointing and recomputation, and offloading data between GPU, host memory, and storage. Systems work combines these methods with parallelism, fused kernels, and memory-aware scheduling to improve throughput as well as fit. The main tension is that memory savings can add computation, communication, or approximation error, so the best trade-off depends on the model, hardware, and training workload.