Kernel optimization is essential for efficient large language model (LLM) training and inference on Ascend NPUs. Existing Ascend kernel benchmarks primarily focus on kernel generation, with less emphasis on the ability of agents to optimize existing implementations under input configurations drawn from real workloads. To address this requirement, we introduce AscendGYM, a benchmark whose initial suite comprises 48 kernels across three difficulty levels. Each kernel is paired with test cases whose data types and tensor shapes are collected from real-world training and inference tasks. Consequently, performance improvements on AscendGYM map directly to the corresponding workloads. In terms of usability, two evaluation modes are supported in AscendGYM: model evaluation and model-harness evaluation. The kernel optimization capabilities of both trained models and custom agents are allowed to be measured. It provides a complete execution environment that closely reproduces how Ascend kernels are invoked in practical pipelines. For model evaluation, it additionally includes a built-in harness equipped with search orchestration and experience accumulation. Finally, we propose a new metric, the Speedup-Token Score (STS), which quantifies performance gains relative to token cost. By jointly considering speedup and cost, STS enables the assessment of practical kernel optimization capability of a model or harness under an acceptable budget. AscendGYM is available at https://github.com/SLAI-AITP/AscendGYM.
Figures & tables
Figure 1: Architecture of AscendForge, integrating resource orchestration, kernel execution, evaluation and feedback, and deployment management for Ascend kernel optimization.
Figure 2: Overview of the AscendGYM workflow.
Figure 3: STS under three token budgets. Solid bars show scores; hatched segments show cost deductions.
Figure 4: CTG at a 50M-token budget. Dots and lines show medians and interquartile ranges over attained tasks; counts give attainment out of 17 tasks in Batch 1.
Figure 5: GM speedup over full search trajectories, where dashed lines mark 1× .
Figure 6: Speedup-threshold attainment ( fastp ) over full search trajectories.
Figure 7: The task pass rate among different combinations.
Figure 8: STS under three token budgets using GPT-5.6 Luna. Solid bars show scores; hatched segments show cost deductions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Initial experience
Online accumulation
Search policy
Cold
None
Strategy + coding
Full Default
Warm
5 strategy lessons
Strategy + coding
Full Default
No lessons
None
None
Agent-search
Code only
None
Code only
Agent-search
Appendix
Table 1: Default configurations.
Component
Configuration or version
Hardware and host system
NPU
Ascend 910C
Runtime device identifier
Ascend910_9392
Host architecture
AArch64
Linux kernel
5.10.0
Runtime and build tools
Appendix
Table 2: Hardware and software environment for AscendForge kernel evaluation.
Figure 9: Per-kernel speedup. Panels show Claude Code, Codex, and OpenCode; dashed lines mark 1× .
Figure 10: Token consumption. Bars show totals in billions (left axis) and per-task means in millions (right axis).
Figure 11: Model interaction rounds. Bars show totals in thousands (left axis) and per-task means (right axis).
Figure 12: Wall-clock time. Bars show summed run hours (left axis) and mean hours per task (right axis).
Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implementation: a host-side tiling program that controls data movement and a kernel program that schedules and pipelines computation. We present AscendOptimizer, an episodic agent that builds missing optimization knowledge from execution itself. For kernel optimization, AscendOptimizer rewinds strong implementations by removing optimizations in a controlled way, then keeps the changes whose removal measurably hurts performance as reusable experience for later rewriting. For host-side optimization, it runs profiling-in-the-loop evolutionary search to find valid, fast tiling and data-movement configurations directly from hardware feedback. This combination lets the agent improve kernel structure and host-side scheduling together. On a benchmark of 101 real AscendC operators, AscendOptimizer achieves a 1.21x geometric-mean speedup over the open-source baseline, and 53.47% of operators run faster than their references. Given a same budget of evaluations per operator, AscendOptimizer consistently outperforms Best-of-N sampling and OpenEvolve in terms of geometric mean speedup, fast_p tail speedup ratios, and overall optimization progress across varying budgets.
Jiehao Wu, Zixiao Huang, Wenhao Li +3
School of Computer Science and Technology, East China Normal University · School of Computer Science and Technology, Tongji University · Shanghai University of International Business and Economics +3
Deep learning inference and training performance depends critically on GPU kernel efficiency. Modern compilers such as PyTorch Inductor automatically generate GPU kernels from high-level model code, but frequently underperform expert-written implementations by wide margins. Recent LLM-assisted optimizers explore these opportunities through kernel generation and refinement; yet translating isolated kernel improvements into model-level gains remains a challenge. We present KernelOPT, a profiling-guided multi-agent system for GPU kernel optimization. It inspects generated Triton kernels and external library calls in compiler artifacts to guide synthesis and refinement, then assembles and evaluates the resulting model. Four stages of validation combine execution checks, multi-seed numerical comparison, model-level checking with a float64 fallback and end-to-end performance gating (γ=1.03). The system supports PyTorch nn.Modules, standalone Triton kernels, and Helion kernels. It is evaluated on all 250 KernelBench problems on NVIDIA H200 which includes 100 single-operator problems (Level1), 100 composite operators (Level2), and 50 model architectures (Level3). KernelOPT achieves geometric mean speedups over torch.compile of 1.40×, 1.15× and 1.07× respectively across L1, L2 and L3. Among problems exceeding 1.03× speedup, the geomeans are 2.54× (36 L1), 1.84× (23 L2), and 1.37× (11 L3). These gains include arithmetic-precision changes and task-specific algebraic simplifications under the evaluated numerical tolerances. The evaluation also reveals discrepancies between isolated kernel improvements and end-to-end performance, motivating model-level measurement alongside kernel-level optimization. Code is available at https://github.com/TorchedHat/KernelOPT.
LLM-based agents for GPU kernel generation are advancing rapidly, yet their progress is fundamentally constrained by the benchmarks they optimize against. Existing benchmarks are poorly aligned with production inference frameworks: they evaluate kernels on a single GPU with synthetic inputs, ignore the surrounding compilation stack, and reward replicating known optimizations rather than discovering new ones. The resulting reward signals are misleading: agents learn to generate kernels that score well in sandboxes but introduce interface incompatibilities, compilation-stack conflicts, and silent correctness degradation when integrated into real systems. We introduce FastKernels, a kernel benchmark built around a minimal set of 46 representative architectures spanning 8 categories, whose kernels collectively subsume those of 96.2% (409/425) of HuggingFace Transformers architectures. FastKernels doubles as a minimalistic, production-grade inference framework that runs at parity with hardened systems such as vLLM and SGLang on mainstream LLM serving and substantially exceeds upstream references on under-served architectures; each task's interface mirrors the corresponding module in the state-of-the-art library for its architecture family, enabling direct deployment of optimized kernels into production codebases. Evaluating state-of-the-art kernel agents on FastKernels, we find that even the strongest agent achieves only 0.94× aggregate speedup over production baselines, with weaker agents at 0.78× and 0.53× -- confirming that benchmark-production misalignment is a critical bottleneck for the field. We release FastKernels as a stepping stone toward kernel agents whose benchmark gains translate directly into production throughput improvements. Code is available at https://github.com/Snowflake-AI-Research/fastkernels