Paper ID: 2310.15074

MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency

Xiaoyun Liu, Divya Saxena, Jiannong Cao, Yuqing Zhao, Penghui Ruan

Neural architecture search (NAS) has gained significant traction in automating the design of neural networks. To reduce the time cost, differentiable architecture search (DAS) transforms the traditional paradigm of discrete candidate sampling and evaluation into that of differentiable super-net optimization and discretization. However, existing DAS methods fail to trade off between model performance and model size. They either only conduct coarse-grained operation-level search, which results in redundant model parameters, or restrictively explore fine-grained filter-level and weight-level units with pre-defined remaining ratios, suffering from excessive pruning problem. Additionally, these methods compromise search quality to save memory during the search process. To tackle these issues, we introduce multi-granularity architecture search (MGAS), a unified framework which aims to discover both effective and efficient neural networks by comprehensively yet memory-efficiently exploring the multi-granularity search space. Specifically, we improve the existing DAS methods in two aspects. First, we balance the model unit numbers at different granularity levels with adaptive pruning. We learn discretization functions specific to each granularity level to adaptively determine the unit remaining ratio according to the evolving architecture. Second, we reduce the memory consumption without degrading the search quality using multi-stage search. We break down the super-net optimization and discretization into multiple sub-net stages, and perform progressive re-evaluation to allow for re-pruning and regrowing of previous units during subsequent stages, compensating for potential bias. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate that MGAS outperforms other state-of-the-art methods in achieving a better trade-off between model performance and model size.

Submitted: Oct 23, 2023