cs.PLFeb 22, 2026

Analyzing Latency Hiding and Parallelism in an MLIR-based AI Kernel Compiler

Authors: Javed Absar, Samarth Narang, Muthu Baskaran

Organizations: Qualcomm Technologies International, Ltd. · Qualcomm Technologies, Inc.

Abstract

AI kernel compilation for edge devices depends on the compiler's ability to exploit parallelism and hide memory latency in the presence of hierarchical memory and explicit data movement. This paper reports a benchmark methodology and corresponding results for three compiler-controlled mechanisms in an MLIR-based compilation pipeline: vectorization (Vec), multi-threading (MT) across hardware contexts, and double buffering (DB) using ping--pong scratchpad buffers to overlap DMA transfers with compute. Using Triton/Inductor-generated kernels, we present an ablation ladder that separates the contribution of Vec, MT, and DB, and we quantify how MT speedup scales with problem size using GELU as a representative activation kernel. The results show that vectorization provides the primary gain for bandwidth-sensitive kernels, MT delivers substantial improvements once scheduling overhead is amortized, and DB provides additional benefit when transfers and compute can be overlapped (i.e., outside the extremes of purely memory-bound or purely compute-bound behavior).

Figures & tables

Explore similar work

CardsList
  1. Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel

    Apr 14, 2026Hongyi Jin, Bohan Hou, Guanjie Wang +18GPU Kernel OptimizationGPU Kernel Generation

  2. Hexagon-MLIR: An AI Compilation Stack For Qualcomm's Neural Processing Units (NPUs)

    Feb 23, 2026Mohammed Javed Absar, Muthu Baskaran, Abhikrant Sharma +22AI Accelerator Inference

  3. AI as a Compiler: Compiling Triton kernels without the Triton compiler

    Sep 29, 2026François Costa, Charly Castes, Thomas Bourgeat +1GPU Kernel OptimizationLarge Language Model-Guided Optimization