Vision Transformer Compression

Vision Transformer (ViT) compression focuses on reducing the computational cost and memory footprint of these powerful but resource-intensive models, while preserving accuracy. Current research explores various techniques, including structured and unstructured pruning of attention heads and MLP layers, low-rank approximation, and model binarization, often employing optimization algorithms like Proximal Policy Optimization or Gaussian process search to find optimal compression strategies across multiple dimensions (attention heads, neurons, and sequence length). These efforts are significant because they enable the deployment of ViTs on resource-constrained devices like mobile phones and embedded systems, expanding their applicability in areas such as autonomous navigation and edge computing.