Fine-Pruning: A Biologically Inspired Algorithm for Personalization of Machine Learning Models
Authors: Joseph Bingham, Saman Zonouz, Dvir Aran
Organizations: Faculty of Biology, Technion - Israel Institute of Technology, Haifa, IL · College of Computing, The Georgia Institute of Technology, Atlanta, GA, USA · Taub Faculty of Computer Science, Technion - Israel Institute of Technology, Haifa, IL
Neural networks have long strived to emulate the learning capabilities of the human brain. While deep neural networks (DNNs) draw inspiration from the brain in neuron design, their training methods diverge from biological foundations. Backpropagation, the primary training method for DNNs, requires substantial computational resources and fully labeled datasets, presenting major bottlenecks in development and application. This work demonstrates that by returning to biomimicry, specifically mimicking how the brain learns through pruning, we can solve various classical machine learning problems while utilizing orders of magnitude fewer computational resources and no labels. Our experiments successfully personalized multiple speech recognition and image classification models, including ResNet50 on ImageNet, resulting in increased sparsity of approximately 70% while simultaneously improving model accuracy to around 90%, all without the limitations of backpropagation. This biologically inspired approach offers a promising avenue for efficient, personalized machine learning models in resource-constrained environments.
Figures & tables
Figure 1: The Fine-Pruning algorithm. First, a general selection of data is used to train the model. Second, this generalized model is downloaded onto various devices, each with its own data domain (with no label) which the general model has not seen. The activation energies are recorded. Third, while on board, the model uses the target data to prune itself making the model smaller and more accurate.
Figure 2: Performance comparison of Fine-Pruning across different datasets and model architectures. a. Accuracy for individuals constructed from the ImageNet dataset, tested on ResNet50. Each group represents a different individual, with bars showing baseline accuracy (blue), Fine-Pruning (red), standard backpropagation (yellow), and SVD (green). Fine-Pruning consistently outperforms other methods, with some individuals seeing dramatic improvements. b. Comparison of accuracy across different individuals using the larger free-spoken-digit dataset on a VGG-7 architecture. The plot shows baseline accuracy (blue), Fine-Pruning (red), backpropagation (yellow), and SVD (green). Fine-Pruning demonstrates consistent improvements across all individuals. c. Results on the CK+ facial emotion dataset using a VGG-19 architecture. The graph compares the baseline model (red), Fine-Pruning (blue), and backpropagation (yellow). Fine-Pruning shows superior performance in personalizing the model for individual subjects’ facial expressions. These results collectively demonstrate the versatility and effectiveness of Fine-Pruning across various domains, datasets, and model architectures. d. Accuracy across different individuals constructed from the ImageNet dataset tested on MobileNetv2 model. Each individual has two classes of the 1000 classes selected from the classes with the highest accuracies in the baseline model.
Figure 3: Performance characteristics of Fine-Pruning compared to traditional methods. a. The accuracy increases until around 70% sparsity, after which it begins to decline. b. Fine-Pruning (blue) begins increasing accuracy with less data compared to backpropagation (orange). c. As the number of target classes increases, the accuracy of Fine-Pruning decreases slightly but remains high. All experiments were performed using ResNet50 on the ImageNet dataset.