Deep Learning Inference

Deep learning inference focuses on efficiently executing pre-trained deep neural networks to make predictions, aiming to optimize speed, energy consumption, and memory usage while maintaining accuracy. Current research emphasizes techniques like model compression (e.g., quantization, pruning, knowledge distillation), adaptive inference (e.g., early exiting), and distributed inference across heterogeneous hardware (including mobile devices, edge servers, and cloud computing) to address resource constraints and latency issues. These advancements are crucial for deploying deep learning in resource-limited environments and enabling real-time applications in diverse fields, from mobile computing and IoT to scientific simulations and medical imaging.

Papers

October 16, 2024

EdgeRL: Reinforcement Learning-driven Deep Learning Model Inference Optimization at Edge
Motahare Mounesan, Xiaojie Zhang, Saptarshi Debroy
Reinforcement Learning Optimization Purpose Extreme Edge Deep Learning Inference CNN Inference Model Based Deep Reinforcement Learning

August 10, 2024

Quantum-secure multiparty deep learning
Kfir Sulimany, Sri Krishna Vadlamani, Ryan Hamerly, Prahlad Iyengar, Dirk Englund
Deep Learning Neural Network Weight Deep Learning Inference Secure Multiparty Computation Practical Quantum Secure Cloud

July 15, 2024

Deep Learning Evidence for Global Optimality of Gerver's Sofa
Kuangdai Leng, Jia Bi, Jaehoon Cha, Samuel Pinilla, Jeyan Thiyagalingam
Global Optimality Linear Neural Network Deep Learning Inference Tabletop Rearrangement

July 1, 2024

Accelerate Intermittent Deep Inference
Ziliang Zhang
Neural Architecture Search Deep Learning Inference Flexible Inference

June 20, 2024

Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices
Li Wang, Liang Li, Lianming Xu, Xian Peng, Aiguo Fei
Scientific Inference Model Compression Inference Time Deep Learning Inference

May 3, 2024

Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls
Sicong Liu, Wentao Zhou, Zimu Zhou, Bin Guo, Minfan Wang, Cheng Fang, Zheng Lin, Zhiwen Yu
Full Potential Common Pitfall Resource Constrained Device Heterogeneous Device Deep Learning Inference Parallel Inference

September 11, 2023

Temporal Patience: Efficient Adaptive Deep Learning for Embedded Radar Data Processing
Max Sponner, Julius Ott, Lorenzo Servadei, Bernd Waschneck, Robert Wille, Akash Kumar
Radar Data Deep Learning Inference Radar Sensor Early Exit Neural Network Adaptive Deep Radar Processing

September 5, 2023

PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference
Marta Andronic, George A. Constantinides
Scientific Inference Field Programmable Gate Array DNN Inference Deep Learning Inference

July 11, 2023

Differentiable Forward Projector for X-ray Computed Tomography
Hyojin Kim, Kyle Champley
Data Driven X Ray Tomographic Reconstruction Deep Learning Inference Back Projection Differentiable Projection

June 24, 2023

Mobile-Cloud Inference for Collaborative Intelligence
Mateen Ulhaq
Inference Latency Deep Learning Inference Collaborative Intelligence Mobile Inference Partial Inference

April 23, 2023

The Case for Hierarchical Deep Learning Inference at the Network Edge
Ghina Al-Atat, Andrea Fresa, Adarsh Prasad Behera, Vishnu Narayanan Moothedath, James Gross, Jaya Prakash Champati
Case Relevance Edge Device Edge AI Edge Network Deep Learning Inference Hierarchical Deep Learning Local Inference Hierarchical Inference

March 31, 2023

FP8 versus INT8 for efficient deep learning inference
Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, Tijmen Blankevoort
Deep Learning Device Inference Personality Classification Deep Learning Inference Artificial Intelligence Inference Bit Weight FP8 Format

February 15, 2023

Toward matrix multiplication for deep learning inference on the Xilinx Versal
Jie Lei, José Flich, Enrique S. Quintana-Ortí
Deep Learning Matrix Multiplication Deep Learning Inference Matrix Multiplication Kernel Neural Engine

January 15, 2023

Adaptive Deep Neural Network Inference Optimization with EENet
Fatih Ilhan, Ka-Ho Chow, Sihao Hu, Tiansheng Huang, Selim Tekin, Wenqi Wei, Yanzhao Wu, Myungjin Lee, Ramana Kompella, Hugo Latapie, Gaowen Liu, Ling Liu
Deep Neural Network Supervised ImageNet DNN Model DNN Inference Deep Learning Inference Multi Exit

October 9, 2022

Deep Learning Inference Frameworks Benchmark
Pierrick Pochelu
Deep Learning Deep Learning Inference Prediction Speed

October 4, 2022

Energy Consumption of Neural Networks on NVIDIA Edge Boards: an Empirical Model
Seyyidahmed Lahmer, Aria Khoshsirat, Michele Rossi, Andrea Zanella
Neural Network Neural Architecture Search Energy Consumption Task Offloading Efficient Architecture Deep Learning Inference

September 29, 2022

Tuning of Mixture-of-Experts Mixed-Precision Neural Networks
Fabian Tschopp
Deep Learning Convolutional Neural Network Inference Speed Mixed Precision Deep Learning Inference OpenCL Code

August 27, 2022

RL-DistPrivacy: Privacy-Aware Distributed Deep Inference for low latency IoT systems
Emna Baccour, Aiman Erbad, Amr Mohamed, Mounir Hamdi, Mohsen Guizani
Deep Neural Network Privacy Preserving Low Latency Deep Learning Inference Privacy Aware Collaborative Deep Neural Network Inference

July 19, 2022

Using Conservation Laws to Infer Deep Learning Model Accuracy of Richtmyer-meshkov Instabilities
Charles F. Jekel, Dane M. Sterbentz, Sylvie Aubry, Youngsoo Choi, Daniel A. White, Jonathan L. Belof
Deep Learning Conservation Law Deep Learning Inference Wave Equation Hydrodynamic Simulation Impact Mechanic Richtmyer Meshkov Instability

July 18, 2022

Accelerating Deep Learning Model Inference on Arm CPUs with Ultra-Low Bit Quantization and Runtime
Saad Ashfaq, MohammadHossein AskariHemmat, Sudhakar Sah, Ehsan Saboori, Olivier Mastropietro, Alexander Hoffman
Deep Learning Deep Learning Model Runtime Analysis Low Bit Quantization Deep Learning Inference Virtual ARM Tensorflow Lite