Structured Pruning

Structured pruning reduces a network by removing hardware-friendly components such as convolutional channels, transformer heads and layers, or mixture-of-experts neurons. Methods rank components using signals such as activation magnitude, gradients, Fisher or Hessian estimates, and task-specific sensitivity, then allocate removals under parameter, memory, or latency budgets. Fine-tuning, distillation, and weight compensation can recover lost performance, while newer approaches target language, vision-language, and generative models as well as smaller vision networks. A central challenge is that high sparsity or fewer FLOPs do not guarantee preserved task quality or real speedups: results depend on calibration data, model structure, recovery cost, and the hardware and kernels used at deployment.