Deep Learning Inference

Deep learning inference focuses on efficiently executing pre-trained deep neural networks to make predictions, aiming to optimize speed, energy consumption, and memory usage while maintaining accuracy. Current research emphasizes techniques like model compression (e.g., quantization, pruning, knowledge distillation), adaptive inference (e.g., early exiting), and distributed inference across heterogeneous hardware (including mobile devices, edge servers, and cloud computing) to address resource constraints and latency issues. These advancements are crucial for deploying deep learning in resource-limited environments and enabling real-time applications in diverse fields, from mobile computing and IoT to scientific simulations and medical imaging.

Papers

July 12, 2022

Look-ups are not (yet) all you need for deep learning inference
Calvin McCarter, Nicholas Dronen
Neural Network Scientific Inference Matrix Multiplication Lookup Table Neural Network Inference Deep Learning Inference

April 8, 2022

C-NMT: A Collaborative Inference Framework for Neural Machine Translation
Yukai Chen, Roberta Chiaro, Enrico Macii, Massimo Poncino, Daniele Jahier Pagliari
Neural Machine Translation Collaborative Inference Deep Learning Inference

January 21, 2022

APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference
Alberto Delmas Lascorz, Mostafa Mahmoud, Andreas Moshovos
Lossless Compression Deep Learning Inference Tensor Core Chip Memory

Deep Learning Inference

Papers

Look-ups are not (yet) all you need for deep learning inference

C-NMT: A Collaborative Inference Framework for Neural Machine Translation

APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference