AI Accelerator Inference

Ask about this topic

Overview

AI accelerator inference concerns how to run trained models on GPUs, TPUs, NPUs, and custom chips with low latency and energy use while preserving output quality. Common approaches include quantization and mixed precision, fused kernels, memory and cache optimization, workload scheduling, and co-designing model algorithms with accelerator dataflows. Research spans large language models and mixture-of-experts serving as well as vision, speech, and embodied AI, often measuring prefill and decoding separately or evaluating real-time edge workloads. A central challenge is that performance depends strongly on the model, inference phase, hardware, and software stack, so gains on one platform may not transfer and can come with accuracy, energy, or portability costs.

Latest papers

CardsList

105 papers7 papers in the last four weeks, down 30% on the four weeks before. 0.0% of all new papers.

  1. Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S

    Mar 24, 2026Ranjith M. S., Akshat Mandloi, Sudarshan KamathTTS SynthesisAI Accelerator Inference

    Smallest AI

  2. EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

    Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative DecodingLLM Inference Acceleration

    College of Computer Science, Chongqing University

  3. Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference

    Mar 6, 2026Ramchand KumaresanNeural Network Hardware AccelerationOn-Device Training

  4. ChatNeuroSim: An LLM Agent Framework for Automated Compute-in-Memory Accelerator Deployment and Optimization

    Mar 4, 2026Ming-Yen Lee, Shimeng YuCompute-in-MemoryElectronic Design Automation

    Georgia Institute of Technology, Atlanta, Georgia, USA

  5. Characterizing VLA Models: Identifying the Action Generation Bottleneck for Edge AI Architectures

    Mar 1, 2026Manoj Vishwanathan, Suvinay Subramanian, Anand RaghunathanVision-Language-Action ModelsEfficient VLA Model Inference

    Google, Mountain View, CA USA · Purdue University, West Lafayette, IN USA

  6. Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

    Feb 27, 2026Athos GeorgiouLLM Inference EfficiencyLLM Inference

  7. The AetherFloat Family: Block-Scale-Free Quad-Radix Floating-Point Architectures for AI Accelerators

    Feb 26, 2026Keita MorisakiStochastic RoundingAI Accelerator Inference

    Independent Hardware/Software Co-Design Researcher Shinagawa City, Tokyo, Japan

  8. RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators

    Feb 24, 2026Xinsheng Tang, Yangcheng Li, Nan Wang +5GPU Kernel OptimizationGPU Kernel Generation

    Alibaba Cloud Computing, Shanghai, China · Alibaba Cloud Computing, Shenzhen, China

  9. Hexagon-MLIR: An AI Compilation Stack For Qualcomm's Neural Processing Units (NPUs)

    Feb 23, 2026Mohammed Javed Absar, Muthu Baskaran, Abhikrant Sharma +22AI Accelerator Inference

    Qualcomm Technologies International, Ltd. · Qualcomm Technologies, Inc. · Qualcomm India Private Limited

  10. CORVET: A CORDIC-Powered, Resource-Frugal Mixed-Precision Vector Processing Engine for High-Throughput AIoT applications

    Feb 22, 2026Sonu Kumar, Mohd Faisal Khan, Mukul Lokhande +1AI Accelerator InferenceEfficient Neural Network Inference

    Centre for Advanced Electronics, IIT Indore. · NSDCS Research Group, Dept. of Electrical Engineering at IIT Indore.

  11. RPU -- A Reasoning Processing Unit

    Feb 20, 2026Matthew Adiletta, Gu-Yeon Wei, David BrooksAI Accelerator InferenceLLM Inference Efficiency

    Harvard University

  12. HAWX: A Hardware-Aware FrameWork for Fast and Scalable ApproXimation of DNNs

    Feb 18, 2026Samira Nazari, Mohammad Saeed Almasi, Mahdi Taheri +4Neural Network Hardware AccelerationNeural Network Optimization

    University of Zanjan, Zanjan, Iran · Brandenburg Technical University, Cottbus, Germany · Tallinn University of Technology, Tallinn, Estonia +1

  13. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

    Nov 11, 2025Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12LLM Inference EfficiencyAI Accelerator Inference

    Stanford University · Together AI

  14. Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

    Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator InferenceLLM Serving

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

  15. PVF:Understanding AI Vulnerability Against SDCs

    May 2, 2024Xun Jiao, Fred Lin, Harish D. Dixit +8AI Accelerator Inference

    Meta Platforms, Inc