Fine-Pruning: A Biologically Inspired Algorithm for Personalization of Machine Learning Models
Authors: Joseph Bingham, Saman Zonouz, Dvir Aran
Organizations: Faculty of Biology, Technion - Israel Institute of Technology, Haifa, IL · College of Computing, The Georgia Institute of Technology, Atlanta, GA, USA · Taub Faculty of Computer Science, Technion - Israel Institute of Technology, Haifa, IL
Neural networks have long strived to emulate the learning capabilities of the human brain. While deep neural networks (DNNs) draw inspiration from the brain in neuron design, their training methods diverge from biological foundations. Backpropagation, the primary training method for DNNs, requires substantial computational resources and fully labeled datasets, presenting major bottlenecks in development and application. This work demonstrates that by returning to biomimicry, specifically mimicking how the brain learns through pruning, we can solve various classical machine learning problems while utilizing orders of magnitude fewer computational resources and no labels. Our experiments successfully personalized multiple speech recognition and image classification models, including ResNet50 on ImageNet, resulting in increased sparsity of approximately 70% while simultaneously improving model accuracy to around 90%, all without the limitations of backpropagation. This biologically inspired approach offers a promising avenue for efficient, personalized machine learning models in resource-constrained environments.
Figures & tables
Figure 1: The Fine-Pruning algorithm. First, a general selection of data is used to train the model. Second, this generalized model is downloaded onto various devices, each with its own data domain (with no label) which the general model has not seen. The activation energies are recorded. Third, while on board, the model uses the target data to prune itself making the model smaller and more accurate.
Figure 2: Performance comparison of Fine-Pruning across different datasets and model architectures. a. Accuracy for individuals constructed from the ImageNet dataset, tested on ResNet50. Each group represents a different individual, with bars showing baseline accuracy (blue), Fine-Pruning (red), standard backpropagation (yellow), and SVD (green). Fine-Pruning consistently outperforms other methods, with some individuals seeing dramatic improvements. b. Comparison of accuracy across different individuals using the larger free-spoken-digit dataset on a VGG-7 architecture. The plot shows baseline accuracy (blue), Fine-Pruning (red), backpropagation (yellow), and SVD (green). Fine-Pruning demonstrates consistent improvements across all individuals. c. Results on the CK+ facial emotion dataset using a VGG-19 architecture. The graph compares the baseline model (red), Fine-Pruning (blue), and backpropagation (yellow). Fine-Pruning shows superior performance in personalizing the model for individual subjects’ facial expressions. These results collectively demonstrate the versatility and effectiveness of Fine-Pruning across various domains, datasets, and model architectures. d. Accuracy across different individuals constructed from the ImageNet dataset tested on MobileNetv2 model. Each individual has two classes of the 1000 classes selected from the classes with the highest accuracies in the baseline model.
Figure 3: Performance characteristics of Fine-Pruning compared to traditional methods. a. The accuracy increases until around 70% sparsity, after which it begins to decline. b. Fine-Pruning (blue) begins increasing accuracy with less data compared to backpropagation (orange). c. As the number of target classes increases, the accuracy of Fine-Pruning decreases slightly but remains high. All experiments were performed using ResNet50 on the ImageNet dataset.
Efficient representation learning is essential for optimal information storage and classification. However, it is frequently overlooked in artificial neural networks (ANNs). This neglect results in networks that can become overparameterized by factors of up to 13, increasing redundancy and energy consumption. As the demand for large language models (LLMs) and their scale increase, these issues are further highlighted, raising significant ethical and environmental concerns. We analyze our previously developed biologically inspired learning rule using information-theoretic concepts, evaluating its efficiency on the MNIST classification task. The proposed rule, which emulates the brain's structural plasticity, naturally prevents overparameterization by optimizing synaptic usage and retaining only the essential number of synapses. Furthermore, it outperforms backpropagation (BP) in terms of efficiency and storage capacity. It also eliminates the need for pre-optimization of network architecture, enhances adaptability, and reflects the brain's ability to reserve 'space' for new memories. This approach advances scalable and energy-efficient AI and provides a promising framework for developing brain-inspired models that optimize resource allocation and adaptability.
Patrick Stricker, Florian Röhrbein, Andreas Knoblauch
KEIM Institute, Albstadt-Sigmaringen University, Germany · Department of Computer Science, Chemnitz University of Technology, Germany
While deep neural networks (DNNs) have achieved remarkable performance in tasks such as image recognition, they often struggle with generalization, learning from few examples, and continuous adaptation - abilities inherent in biological neural systems. These challenges arise due to DNNs' failure to emulate the efficient, adaptive learning mechanisms of biological networks. To address these issues, we explore the integration of neurobiologically inspired assumptions in neural network learning. This study introduces a biologically inspired learning rule that naturally integrates neurobiological principles, including sparsity, lognormal weight distributions, and adherence to Dale's law, without requiring explicit enforcement. By aligning with these core neurobiological principles, our model enhances robustness against adversarial attacks and demonstrates superior generalization, particularly in few-shot learning scenarios. Notably, integrating these constraints leads to the emergence of biologically plausible neural representations, underscoring the efficacy of incorporating neurobiological assumptions into neural network design. Preliminary results suggest that this approach could extend from feature-specific to task-specific encoding, potentially offering insights into neural resource allocation for complex tasks.
Patrick Inoue, Florian Röhrbein, Andreas Knoblauch
KEIM Institute, Albstadt-Sigmaringen University, Germany · Department of Computer Science, Chemnitz University of Technology, Germany
Deep learning models achieve state-of-the-art image classification but face deployment challenges due to computational costs and energy demands. We propose a lightweight training strategy that adapts normalization layers of the model to the new domain and decouples feature extraction from classifier optimization, reducing overhead by precomputing features only once. A redesigned classifier head with margin-based weighted loss further minimizes ambiguity without end-to-end backpropagation. Evaluated across four CNN architectures (ResNet18, ResNet50, MobileNet, DenseNet121), three Transformer models (ViT, Swin and DeiT) and three medical datasets (Brain Cancer MRI, BreakHis and PatchCamelyon), our approach significantly reduces the required training time with only a marginal accuracy trade-off, often matching or surpassing baseline performance. This efficiency translates to reducing CO2 by orders of magnitude, offering a practical and environmentally sustainable solution for resource-constrained clinical or prototyping environments.
Daniel Vila-Cruz, Laura Morán-Fernández, Verónica Bolón-Canedo