As high-performance computing and AI workloads become increasingly dependent on GPUs, maintaining high performance across rapidly evolving hardware generations has become a major challenge. Developers often spend months tuning scientific applications to fully exploit new architectures, navigating a complex optimization space that spans algorithm design, source implementation, compiler flags and pass sequences, and kernel launch parameters. Existing approaches can effectively search parts of this space in isolation, such as launch configurations or compiler settings, but optimizing across the full space still requires substantial human expertise and iterative manual effort. In this paper, we present Record-Remix-Replay (R^3), a hierarchical optimization framework that combines LLM-driven evolutionary search, Bayesian optimization, and record-replay compilation techniques to efficiently explore GPU kernel optimizations from source-level implementation choices down to compiler pass ordering and runtime configuration. By making candidate evaluation fast and scalable, our approach enables practical end-to-end search over optimization dimensions that are typically treated separately. We show that Record-Remix-Replay can optimize full scientific applications better than traditional approaches over kernel parameters and compiler flags, while also being nearly an order of magnitude faster than modern evolutionary search approaches.
Figures & tables
Fig. 1: Overview of MAP-Elites evolution as in AlphaEvolve/OpenEvolve. 1 Prompts are constructed from the MAP-Elites population database, 2 sent to an LLM randomly selected from an ensemble of LLMs, and 3 evaluated by compiling and running the code. The population database is then updated according to the MAP-Elites algorithm. Multiple parallel controllers execute this loop concurrently to increase throughput.
Source-level
Compiler-level
Runtime
System
Eval.
Work
Unstruct. rewrite
DSL / template
Flags / params
Pass order
Launch cfg.
Env. / resources
Required Execution
Cost
Kernel parameter auto-tuners
CLTune; Kernel Tuner [ 4 , 15 ]
✓
✓
∘
Kernel
Low
Kernel Launcher [ 3 ]
✓
✓
Kernel
Low
DSL auto-schedulers
TVM / Ansor / MetaSchedule [ 16 , 17 , 18 ]
✓
∘
✓
Kernel / Graph
Med.
TABLE I: Representative related work in GPU kernel auto-tuning and optimization. ✓ = first-class support; ∘ = technically possible but indirect / not the primary design point. Evaluation cost is qualitative and reflects the typical inner-loop tuning cost: Low = isolated kernel / replay / JIT evaluation, Med. = operator or graph-level tuning, High = full-application.
Fig. 2: Overview of our Record-Remix-Replay framework. 1 First, the prefix-aware prompt sampler is used to construct prompts that optimize prefix cache hits. 2 An LLM is selected intelligently based on estimated generation time to maximize GPU utilization and is used to optimize the GPU kernel. 3 LLVM IR is generated for the optimized kernel and sent to the replay server, which replays kernels and runs parallel Bayesian optimization to find the best runtime across compiler passes and launch parameters. The population database is then updated according to the MAP-Elites algorithm.
Fig. 3: The breakdown in times spent in generation, compiling, and evaluation during evolution with each progressive optimization used. Results are shown for 1 sequential evolve process using gpt-oss-120b with high reasoning effort on the QUDA application. We see our proposed optimizations have moved evolutionary optimization from evaluation-bound to LLM inference-bound.
Fig. 4: Comparison of time spent generating outputs with and without prefix-aware prompts across a 250 iteration evolution run. We see a reduction in median generation time for both a commercial (gpt-5-mini) and local (gpt-oss-120b) model. Notably, most commercial model providers offer discounts for prefix-cache hits, leading to lower overall costs during evolution.
Fig. 5: Comparison of OpenEvolve’s LLM selection algorithm to R3 ’s runtime-aware LLM selection. Whereas increasing the number of parallel evolution processes w.r.t. the number of GPUs can increase utilization and throughput, it comes at the cost of increasing stale MAP-Elites database reads and slowing convergence. R3 ’s runtime-aware LLM selection increases utilization and throughput over OpenEvolve for the same number of processes.
Fig. 6: Comparison of kernel speedups across the approaches and applications on an MI300A. Times are shown for all kernels in an application (Lulesh 15, MiniFE 7, s3d 54, and miniWeather 7) and dots represent outliers. R3 yields higher median and max speedups over BO and OpenEvolve.
Fig. 7: Comparison of kernel speedups across the approaches and applications on an H100. Times are shown for all kernels in an application and dots represent outliers. R3 yields higher median and max speedups over baselines.
Fig. 8: Comparison of absolute times from kernels in the miniWeather application. The y-axis encodes kernel time after R3 optimization and the x-axis encodes kernel time after optimization from record-replay + BO (left plot) and OpenEvolve (right plot). Points under the line are faster with R3 .
Fig. 9: Comparison of final achieved speedup versus the time-to-solution. The vertical axis shows speedup over the original application and the horizontal axis shows time-to-solution in minutes (note log scale). While R3 is slower than BO with record-replay, it produces consistently better tuning results. Furthermore, R3 is both faster and yields better speedups than OpenEvolve.
GPU kernel optimization challenges LLMs beyond standard coding tasks, as it requires an understanding of hardware architecture, parallel computing optimization strategies, and profiling outputs. However, most existing approaches leveraging LLMs for kernel generation apply standard prompting and feedback loops, considering hardware only through profiling feedback. We introduce KernelFoundry, an evolutionary framework that efficiently explores the space of GPU kernels through (1) MAP-Elites quality diversity search with kernel-specific behavioral dimensions to sustain exploration; (2) meta-prompt evolution that co-evolves prompts with kernels to uncover task-specific optimization strategies, and (3) a template-based parameter optimization approach to tune kernels to inputs and hardware. We evaluate this framework on Kernel-Bench, robust-kbench and custom tasks, generating SYCL kernels as a cross-platform GPU programming paradigm, and CUDA kernels for comparison to prior work. Our approach consistently outperforms the baseline methods and achieves an average speedup of 2.3 on KernelBench for SYCL. Moreover, KernelFoundry is implemented as a distributed framework with remote access to diverse hardware, allowing quick benchmarking and featuring a flexible user input layer to support kernel generation for a wide range of real use cases beyond benchmarking.
Nina Wiedemann, Quentin Leboutet, Michael Paulitsch +2
High-performance GPU kernels underpin modern deep learning and scientific computing. As workloads become increasingly diverse and GPU hardware evolves rapidly, developing efficient methods for automated GPU kernel generation and optimization has become increasingly important. Existing LLM-based methods typically optimize within a fixed implementation space, limiting either optimization flexibility or search efficiency. We propose HIERA, a hierarchical search-space planning framework for GPU kernel optimization. HIERA constructs contract-augmented task specifications, selects an appropriate implementation space across PyTorch operators, CUDA libraries, and custom CUDA kernels, and uses profiling feedback and expert knowledge to guide structured iterative refinement. Experiments on KernelBench across multiple various workload levels and base LLMs show that HIERA delivers stronger overall implementation validity, sample efficiency, and optimization performance than existing training-free methods, while remaining competitive with the training-based CUDA-L1 without additional model training. A case study on a specialized stencil operator from scientific computing further achieves a 1.53× speedup over cuDNN, demonstrating the potentiality of the general framework beyond standard machine-learning workloads.
Jinghao Wang, Qiqi Gu, Chenpeng Wu +3
Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University
Optimizing GPU kernels is critical for efficient modern machine learning systems yet remains challenging due to the complex interplay of design factors and rapid hardware evolution. Existing automated approaches typically treat Large Language Models (LLMs) merely as stochastic code generators within heuristic-guided evolutionary loops. These methods often struggle with complex kernels requiring coordinated, multi-step structural transformations, as they lack explicit planning capabilities and frequently discard promising strategies due to inefficient or incorrect intermediate implementations. To address this, we propose Search via Co-Evolving World Model and build K-Search based on this method. By replacing static search heuristics with a co-evolving world model, our framework leverages LLMs' prior domain knowledge to guide the search, actively exploring the optimization space. This approach explicitly decouples high-level algorithmic planning from low-level program instantiation, enabling the system to navigate non-monotonic optimization paths while remaining resilient to temporary implementation defects. We evaluate K-Search on diverse, complex kernels from FlashInfer, including GQA, MLA, and MoE kernels. Our results show that K-Search significantly outperforms state-of-the-art evolutionary search methods, achieving an average 2.10x improvement and up to a 14.3x gain on complex MoE kernels. On the GPUMode TriMul task, K-Search achieves state-of-the-art performance on H100, reaching 1030us and surpassing both prior evolution and human-designed solutions.