Paper ID: 2312.08053

Kimad: Adaptive Gradient Compression with Bandwidth Awareness

Jihao Xin, Ivan Ilin, Shunkang Zhang, Marco Canini, Peter Richtárik

In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.

Submitted: Dec 13, 2023