Memory-Efficient Fine-Tuning

This field studies how to adapt pretrained models when memory for weights, activations, gradients, and optimizer states is limited, including on consumer GPUs and edge devices. Common approaches include LoRA and quantized adapters, selective or sparse updates, activation checkpointing and offloading, and zeroth-order optimization that avoids storing backpropagated gradients. Work also targets full fine-tuning and long-context training through streamed parameter updates, compressed states, and tiered memory systems. A central tension is that fewer trainable parameters do not always reduce activation memory, and memory savings can trade off against accuracy, speed, energy use, or implementation complexity.