High-performance GPU kernels underpin modern deep learning and scientific computing. As workloads become increasingly diverse and GPU hardware evolves rapidly, developing efficient methods for automated GPU kernel generation and optimization has become increasingly important. Existing LLM-based methods typically optimize within a fixed implementation space, limiting either optimization flexibility or search efficiency. We propose HIERA, a hierarchical search-space planning framework for GPU kernel optimization. HIERA constructs contract-augmented task specifications, selects an appropriate implementation space across PyTorch operators, CUDA libraries, and custom CUDA kernels, and uses profiling feedback and expert knowledge to guide structured iterative refinement. Experiments on KernelBench across multiple various workload levels and base LLMs show that HIERA delivers stronger overall implementation validity, sample efficiency, and optimization performance than existing training-free methods, while remaining competitive with the training-based CUDA-L1 without additional model training. A case study on a specialized stencil operator from scientific computing further achieves a 1.53× speedup over cuDNN, demonstrating the potentiality of the general framework beyond standard machine-learning workloads.
Figures & tables
Figure 1: Comparison of prior methods operating within fixed implementation spaces and our proposed HIERA ’s workload-aware cross-granularity planning.
Figure 2: Overview of HIERA. The framework constructs a contract-augmented task specification, performs hierarchical planning over implementation spaces and optimization directions, and iteratively refines candidate implementations using evaluation and profiling feedback.
Component
Configuration
GPU
NVIDIA A100-PCIE-40GB
CPU
Intel Xeon Gold 6430
Operating system
Ubuntu 20.04.6 LTS
NVIDIA driver
565.57.01
CUDA Toolkit
12.8
Python
3.12.13
Table 1: Hardware and software environment.
Figure 3: Limited-budget performance on KernelBench. At each budget B , we report the best valid implementation among the first B generated candidates. All points are extracted from the same generation trajectories.
Base LLM
Method
Level 1
Level 2
Level 3
fast0
fast1
fast2
fast0
fast1
fast2
fast0
fast1
fast2
DeepSeek-V3.2
HIERA
91
56
22
90
35
9
64
42
10
KernelBench-Caesar
77
17
3
80
34
5
46
12
2
CUDAForge
90
22
6
78
27
15
60
32
10
Qwen3.6-Plus
HIERA
97
68
18
99
62
20
70
42
10
KernelBench-Caesar
91
21
6
90
40
13
64
26
6
Table 2: Comparison with baselines under different base LLMs. Results are percentages. Bold indicates the best result among methods using the same base LLM. † indicates that CUDA-L1 does not vary with the base LLM setting and is therefore reported once.
Figure 4: Distribution of the best verified speedup achieved by three fixed implementation spaces and HIERA on 90 KernelBench tasks sampled from Levels 1–3.
Figure 5: Ablation results across KernelBench Levels 1–3. Blue bars denote the complete HIERA framework. Orange bars denote the corresponding ablated variant.
Figure 6: Search progress compared with cuDNN on the 2D box stencil workload ( R=3 , 49 stencil points).
Improving GPU kernel efficiency is crucial for efficient AI systems. Recent work has explored leveraging large language models (LLMs) for GPU kernel generation and optimization. However, existing LLM-based kernel optimization pipelines typically rely on opaque, implicitly learned heuristics within the LLMs to determine optimization strategies. This leads to inefficient trial-and-error and weakly interpretable optimizations. Our key insight is to replace implicit heuristics with expert optimization knowledge that is aware of task trajectories. Specifically, we present KernelClaw, a multi-agent framework with a dual-level memory architecture. KernelClaw operates by coordinating agents with long-term memory of reusable expert knowledge and short-term memory to prevent repetitive backtracking. On KernelBench Levels 1-3, KernelClaw achieves a 100% success rate and average speedups of 6.98x, 7.89x, and 1.94x over Torch Eager on Levels 1, 2, and 3, respectively, outperforming prior baselines. Code is available at https://github.com/0satan0/KernelMem/.
Qitong Sun, Jun Han, Tianlin Li +6
School of Computer Science and Engineering, Beihang University, China · Zhejiang Lab, China · Nanyang Technological University, Singapore
Writing high-performance GPU kernels is among the most labor-intensive tasks in machine learning systems engineering. We present AutoKernel, an open-source framework that applies an autonomous agent loop to GPU kernel optimization for arbitrary PyTorch models. Given a model, AutoKernel profiles it to identify computational bottlenecks, ranks them by Amdahl's law impact, and iteratively refines Triton or CUDA C++ kernel implementations through hundreds of experiments without human intervention. A five-stage correctness harness covering smoke tests, shape sweeps, numerical stability, determinism verification, and edge-case coverage ensures every candidate kernel is validated before any speedup is recorded. The system comprises over 9,000 lines of Python, 18 starter kernel implementations across two backends, a six-tier optimization playbook, and integration with the KernelBench benchmark suite. AutoKernel covers nine kernel types spanning the dominant operations in modern transformer architectures. On an NVIDIA H100, our Triton kernels outperform both PyTorch eager and torch.compile (max-autotune) on the majority of tested configurations: 5.29x over eager on RMSNorm, 2.82x on softmax, and 2.21x on cross-entropy, while beating torch.compile by 2.83x, 3.44x, and 2.94x respectively. In community deployment, an AutoKernel-optimized kernel achieved first place on the vectorsum_v2 B200 leaderboard. The full system is available at https://github.com/RightNow-AI/autokernel.
GPU kernel optimization challenges LLMs beyond standard coding tasks, as it requires an understanding of hardware architecture, parallel computing optimization strategies, and profiling outputs. However, most existing approaches leveraging LLMs for kernel generation apply standard prompting and feedback loops, considering hardware only through profiling feedback. We introduce KernelFoundry, an evolutionary framework that efficiently explores the space of GPU kernels through (1) MAP-Elites quality diversity search with kernel-specific behavioral dimensions to sustain exploration; (2) meta-prompt evolution that co-evolves prompts with kernels to uncover task-specific optimization strategies, and (3) a template-based parameter optimization approach to tune kernels to inputs and hardware. We evaluate this framework on Kernel-Bench, robust-kbench and custom tasks, generating SYCL kernels as a cross-platform GPU programming paradigm, and CUDA kernels for comparison to prior work. Our approach consistently outperforms the baseline methods and achieves an average speedup of 2.3 on KernelBench for SYCL. Moreover, KernelFoundry is implemented as a distributed framework with remote access to diverse hardware, allowing quick benchmarking and featuring a flexible user input layer to support kernel generation for a wide range of real use cases beyond benchmarking.
Nina Wiedemann, Quentin Leboutet, Michael Paulitsch +2