DNN Inference

DNN inference focuses on efficiently executing pre-trained deep neural networks, aiming to minimize latency, energy consumption, and memory footprint while maintaining accuracy. Current research emphasizes optimizing inference across diverse hardware platforms (e.g., mobile devices, microcontrollers, edge servers, and cloud), exploring techniques like model compression, adaptive batching, workload partitioning, and mixed-precision computation. These advancements are crucial for deploying DNNs in resource-constrained environments and improving the performance and sustainability of AI applications across various domains, from mobile computing to autonomous systems.

Papers

November 6, 2024

Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision
Dinithi Jayasuriya, Nastaran Darabi, Maeesha Binte Hashem, Amit Ranjan Trivedi
Low Precision Neural Network Inference DNN Inference High Precision Accuracy Loss Universal Polarization Neural Polarization

October 2, 2024

Learning the Optimal Path and DNN Partition for Collaborative Edge Inference
Yin Huang, Letian Zhang, Jie Xu
Deep Neural Network DNN Model Optimal Path Collaborative Inference DNN Inference Adversarial Linear Contextual Bandit

May 9, 2024

Anole: Adapting Diverse Compressed Models For Cross-Scene Prediction On Mobile Devices
Yunzhe Li, Hongzi Zhu, Zhuohong Deng, Yunlong Cheng, Liang Zhang, Shan Chang, Minyi Guo
Deep Neural Network Mobile Device DNN Inference Compressed Model Cross Scene

May 1, 2024

vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs
Size Zheng, Renze Chen, Meng Li, Zihao Ye, Luis Ceze, Yun Liang
Neural Network Low Power DNN Inference Dynamic Memory Kernel Optimization

March 23, 2024

Carbon Intensity-Aware Adaptive Inference of DNNs
Jiwan Jung
DNN Framework DNN Inference Accurate Model Carbon Emission Adaptive Inference Carbon Figure

March 13, 2024

Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networks
Paul Ardis, Arjuna Flenner
Deep Neural Network High Uncertainty Anticipation Black Box DNN Inference Confidence Aware Distribution Detection Task Justification Theory

March 2, 2024

Beyond Inference: Performance Analysis of DNN Server Overheads for Computer Vision
Ahmed F. AbouElhamayed, Susanne Balle, Deshanand Singh, Mohamed S. Abdelfattah
Deep Neural Network Computer Vision Scientific Inference Vision Task Performance Analysis DNN Inference

January 30, 2024

January 23, 2024

Full-Stack Optimization for CAM-Only DNN Inference
João Paulo C. de Lima, Asif Ali Khan, Luigi Carro, Jeronimo Castrillon
Supervised ImageNet Associative Memory DNN Inference Compute in Memory Memory Accelerator Racetrack Memory

December 16, 2023

Opara: Exploiting Operator Parallelism for Expediting DNN Inference on GPUs
Aodong Chen, Fei Xu, Li Han, Yuan Dong, Li Chen, Zhi Zhou, Fangming Liu
Single GPU DNN Inference Inference Workload Intensive Operator

October 16, 2023

Adaptive Workload Distribution for Accuracy-aware DNN Inference on Collaborative Edge Platforms
Zain Taufique, Antonio Miele, Pasi Liljeberg, Anil Kanduri
DNN Inference Collaborative Edge Workload Distribution Network Heterogeneity

September 5, 2023

PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference
Marta Andronic, George A. Constantinides
Scientific Inference Field Programmable Gate Array DNN Inference Deep Learning Inference

August 14, 2023

Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling
Lequn Chen, Weixin Deng, Anirudh Canumalla, Yu Xin, Danyang Zhuo, Matthai Philipose, Arvind Krishnamurthy
DNN Framework DNN Model Dynamic Scheduling DNN Inference Phantom 2D Accelerator

August 10, 2023

Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips
Ismet Dagli, Mehmet Belviranli
Diverse Set DNN Inference Chip Memory Heterogeneous System on Chip Layer by Layer

June 29, 2023

Performance Analysis of DNN Inference/Training with Convolution and non-Convolution Operations
Hadi Esmaeilzadeh, Soroush Ghodrati, Andrew B. Kahng, Sean Kinzer, Susmita Dey Manasi, Sachin S. Sapatnekar, Zhiang Wang
Training Data Direct Convolution Convolution Layer Performance Analysis Convolution Operation DNN Inference Systolic Array

June 28, 2023

DNA-TEQ: An Adaptive Exponential Quantization of Tensors for DNN Inference
Bahareh Khabbazan, Marc Riera, Antonio González
Quantization Operator Tensor Network Biological Sequence Tensor Type Calculus DNN Inference Uniform Quantization Adaptive Quantization

June 22, 2023

MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge
Sokratis Nikolaidis, Stylianos I. Venieris, Iakovos S. Venieris
Multi Task Cascade Model DNN Inference Consumer Generated Medium Aware Job Cascade System

June 8, 2023

Precision-aware Latency and Energy Balancing on Multi-Accelerator Platforms for DNN Inference
Matteo Risso, Alessio Burrello, Giuseppe Maria Sarda, Luca Benini, Enrico Macii, Massimo Poncino, Marian Verhelst, Daniele Jahier Pagliari
Low Latency Hardware Accelerator DNN Inference Hardware Aware Multi Accelerator Heterogeneous System on Chip

May 23, 2023

Negative Feedback Training: A Novel Concept to Improve Robustness of NVCIM DNN Accelerators
Yifan Qin, Zheyu Yan, Wujie Wen, Xiaobo Sharon Hu, Yiyu Shi
Native Robustness DNN Accelerator DNN Inference Non Volatile Memory DNN Robustness Novel Concept Device Variation Negative Training

DNN Inference

Papers

Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision

Learning the Optimal Path and DNN Partition for Collaborative Edge Inference

Anole: Adapting Diverse Compressed Models For Cross-Scene Prediction On Mobile Devices

vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs

Carbon Intensity-Aware Adaptive Inference of DNNs

Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networks

Beyond Inference: Performance Analysis of DNN Server Overheads for Computer Vision

Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

Full-Stack Optimization for CAM-Only DNN Inference

Opara: Exploiting Operator Parallelism for Expediting DNN Inference on GPUs

Adaptive Workload Distribution for Accuracy-aware DNN Inference on Collaborative Edge Platforms

PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference

Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling

Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips

Performance Analysis of DNN Inference/Training with Convolution and non-Convolution Operations

DNA-TEQ: An Adaptive Exponential Quantization of Tensors for DNN Inference

MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge

Precision-aware Latency and Energy Balancing on Multi-Accelerator Platforms for DNN Inference

Negative Feedback Training: A Novel Concept to Improve Robustness of NVCIM DNN Accelerators