Overview

AI accelerator inference concerns how to run trained models on GPUs, TPUs, NPUs, and custom chips with low latency and energy use while preserving output quality. Common approaches include quantization and mixed precision, fused kernels, memory and cache optimization, workload scheduling, and co-designing model algorithms with accelerator dataflows. Research spans large language models and mixture-of-experts serving as well as vision, speech, and embodied AI, often measuring prefill and decoding separately or evaluating real-time edge workloads. A central challenge is that performance depends strongly on the model, inference phase, hardware, and software stack, so gains on one platform may not transfer and can come with accuracy, energy, or portability costs.

Latest papers

CardsList

114 papers12 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. dit.xpu : Taming DiT Sparse Inference on Mobile SoCs

    Nov 14, 2026Jiayu Chen, Jinye Hu, Maoliang Li +8Diffusion Model Inference AccelerationOn-Device Inference

  2. Efficient VLA Inference via Retrieval-Based Speculation and Hardware Acceleration

    Nov 8, 2026Zhengyang Cheng, He Zhu, Zeyu Guo +6Efficient VLA Model InferenceAI Accelerator Inference

  3. The World Model Planning Accelerator (WMPA): Hardware-Algorithm Co-Design for Embedded AI in Robotics

    Nov 6, 2026Prerana Ramkumar, Suhas Suresh Bharadwaj, Fazal HameedWorld Model-Based PlanningWorld Models

  4. CAMP: Context-Aware Mapping and Placement for LLMs on 3D-IMC Arrays

    Nov 5, 2026Orhun Görkem, Athanasios Vasilopoulos, Georgios Pagonis +6LLM Inference AccelerationAI Accelerator Inference

  5. Where Is VLA Inference Time Spent? Bottlenecks and Optimization Interactions Across GPUs

    Nov 4, 2026Muhammad Danish Waseem, Ahmed Ali-Eldin HassanEfficient VLA Model InferenceInference-Time Optimization

  6. RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

    Oct 30, 2026Wentao Ni, Chihao Yu, Hong Cai +4Mixture-of-Experts InferenceLLM Inference Acceleration

  7. Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs

    Oct 26, 2026Yilong Li, Chengpo Yan, Yijing Zeng +3Efficient VLM InferenceVLM Inference

  8. AIPC: Evidence-Guided Agent Workflows for PyTorch Project Deployment

    Oct 26, 2026Zhanwei Wu, Jianhao Su, Brian Zhang +11AI Accelerator InferenceLLM Agent Workflow Optimization

  9. DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium

    Oct 16, 2026Bin Ma, Min Yu, Vijay Kodamalla +3Diffusion Model Inference AccelerationDiffusion Model Serving

  10. DynaTE: Accelerating Diffusion LLMs via Dynamic Token Execution

    Oct 8, 2026Minghan Jiang, Jiayi Wang, Shuaiting Li +2AI Accelerator InferenceDiffusion Language Model Inference

  11. Agentic Design Space Exploration for Joint Hardware Configuration Selection and Mapping of AI Inference Workloads on Heterogeneous Edge SoCs

    Oct 5, 2026Geetha Prasuna Yarramneni, Surya Selvam, Wilfried Haensch +1Hardware-Software Co-DesignElectronic Design Automation

  12. LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

    Sep 30, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5AI Accelerator InferenceLLM Inference Acceleration

  13. Accelerator Choice Is Not Enough: AlphaFold2 Inference on Cloud TPUs

    Sep 28, 2026Lorenzo Pazienza, Ihab El BaniAI Accelerator InferenceEfficient Inference

  14. Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

    Sep 28, 2026Zhengxiang Huang, Shengheng Chen, Chaoyue Niu +6AI Accelerator InferenceOn-Device Language Model Inference

  15. MorphAtt: A Neuromorphic Accelerator for Efficient Multi-Head Attention Processing in Spiking Vision Transformers

    Sep 27, 2026Rachmad Vidya Wicaksana Putra, Amirhesam Jafari Rad, Muhammad ShafiqueAI Accelerator InferenceSpiking Transformers

  16. Programming AMD XDNA NPUs with Open-source Compiler Tools: A FlashAttention Case Study

    Sep 18, 2026Erwei Wang, Ephrem Wu, Victor J. B. Jung +4Roofline ModelingAI Accelerator Inference

  17. MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration

    Sep 17, 2026Yuan Liao, Jae-sun SeoAI Accelerator InferenceVLM Quantization

  18. OptiPrime: Optimizing Private Inference through Protocol-Hardware Co-design

    Sep 15, 2026Jiangrui Yu, Ye Yu, Si Chen +5Efficient Neural Network InferenceSecure Multi-Party Computation

  19. A 25-μμs/inf Event-driven Graph Neural Network Processor with Spatiotemporal Caching and Spline Convolution for Ultra-low-latency AI at the Edge

    Sep 14, 2026Adrian Kneip, Martin Lefebvre, Daniel Gehrig +4Event-Based VisionAI Accelerator Inference

  20. NeuroFlex: Lossless Element-Level ANN-SNN Co-Execution for Efficient Sparse Inference

    Sep 12, 2026Varun Manjunath, Pranav Ramesh, Gopalakrishnan SrinivasanEfficient Neural Network InferenceAI Accelerator Inference

  21. Vision-centric generative AI models: A software-hardware perspective

    Aug 27, 2026Eleni Tselepi, Cristian Sestito, Shady Agwa +1Generative ModelingHardware-Software Co-Design