Efficient Deep Learning

Efficient deep learning focuses on developing and deploying deep neural networks that require fewer computational resources and memory, while maintaining high accuracy. Current research emphasizes techniques like model compression (pruning, quantization, low-rank approximation), optimized architectures (e.g., custom designs surpassing pre-trained models like MobileNetV2), and algorithmic improvements (e.g., inverted activations for reduced memory footprint in transformers and LSTMs). This field is crucial for expanding the applicability of deep learning to resource-constrained devices (mobile, IoT) and addressing environmental concerns related to energy consumption, while also mitigating security vulnerabilities associated with publishing smaller, more accessible models.

Papers

November 12, 2024

LAUREL: Learned Augmented Residual Layer
Gaurav Menghani, Ravi Kumar, Sanjiv Kumar
Convolutional Neural Network Supervised ImageNet Residual Connection Safe LoRA Efficient Deep Learning

July 22, 2024

Inverted Activations: Reducing Memory Footprint in Neural Network Training
Georgii Novikov, Ivan Oseledets
Input Tensor Efficient Deep Learning

May 28, 2024

Modeling Long Sequences in Bladder Cancer Recurrence: A Comparative Evaluation of LSTM,Transformer,and Mamba
Runquan Zhang, Jiawen Jiang, Xiaoping Shi
Long Short Term Memory Recurrent Event Recurrence Prediction Recurrent Model Efficient Deep Learning Bladder Cancer

January 26, 2024

Do deep neural networks utilize the weight space efficiently?
Onur Can Koyun, Behçet Uğur Töreyin
Convolutional Neural Network Deep Neural Network Deep Learning Model Parameter Efficient Weight Space Efficient Deep Learning

March 22, 2023

Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training
Xinwei Ou, Zhangxin Chen, Ce Zhu, Yipeng Liu
Deep Learning Low Rank Low Rank Approximation Balancing Strategy Faster Training Efficient Deep Compression Performance Efficient Deep Learning

December 28, 2022

Publishing Efficient On-device Models Increases Adversarial Vulnerability
Sanghyun Hong, Nicholas Carlini, Alexey Kurakin
Deep Neural Network Adversarial Vulnerability Device Model Efficient Deep Learning

May 14, 2022

Efficient Deep Learning Methods for Identification of Defective Casting Products
Bharath Kumar Bolla, Mohan Kingam, Sabeesh Ethiraj
Pre Trained Person Identification Lightweight Model Surface Defect Performing Model Efficient Deep Learning

April 25, 2022

Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
Han Cai, Ji Lin, Yujun Lin, Zhijian Liu, Haotian Tang, Hanrui Wang, Ligeng Zhu, Song Han
Deep Learning Deep Neural Network Financial Application Neural Architecture Search NCD Method System Description Mobile Device Device Training Efficient Neural Network Architecture Efficient Deep Learning

February 1, 2022

Local Feature Matching with Transformers for low-end devices
Kyrylo Kolodiazhnyi
Transformer Megatron Decepticons Image Pair Local Feature Resource Constrained Device Local Feature Matching Efficient Deep Learning