CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe

Tara Saba; Zhiyang Chen; Jikai Jason Li; Anne Ouyang; Xujie Si; Fan Long

cs.LGApr 1, 2026

CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe

Authors: Tara Saba, Zhiyang Chen, Jikai Jason Li, Anne Ouyang, Xujie Si, Fan Long

Organizations: Department of Computer Science, University of Toronto · Standard Kernel

Abstract

High-performance GPU kernels are critical to modern machine learning systems, yet developing them remains a manual, expert-driven process. Recent work has explored using LLMs to automate kernel generation, but generated kernels still fall short of carefully tuned references on standardized benchmarks. We present CuTeGen, an agentic GPU kernel synthesis framework that treats kernel development as a structured generate-test-refine workflow over the CuTe abstraction layer. Two design choices distinguish CuTeGen from prior work: targeting CuTe rather than raw CUDA, which exposes performance-critical structures such as tiling and data movement while remaining stable enough for iterative refinement, and a delayed profiling schedule that withholds low-level performance feedback until the kernel's high-level structure has stabilized. On the 209 tasks of KernelBench Level-1 and Level-2, CuTeGen achieves an average speedup of 1.71 $\times$ over PyTorch and outperforms the prior agentic baseline CudaForge (0.89 $\times$ ) at comparable per-task generation cost. Code available at https://github.com/taratt/cutegen.git

Figures & tables

Figure 1: Detailed view of the correctness assurance component. Generated kernels are compiled and validated against reference outputs, and structured patches are applied iteratively until correctness is achieved or the generation budget is exhausted.

Figure 2: Detailed view of the optimization component. Profiling feedback is introduced according to a workload-dependent strategy to guide performance-aware refinement.

Tool	Speedup $\uparrow$	Low Precision $\downarrow$	Correct $\uparrow$	Fast1 $\uparrow$	Speedup On Fast1 $\uparrow$	Best $\uparrow$	Cost (USD) $\downarrow$
CudaForge	0.89 $\times$	24%	63%	22%	1.45 $\times$	11%	1.41
CuTeGen	1.71 $\times$	0%	87%	36%	2.16 $\times$	34%	1.69

Table 1: Comparison of CuTeGen and CudaForge on KernelBench Level-1 and Level-2 workloads. We report average speedup, precision lowering rate (%), correctness (%), Fast1 (%), speedup on Fast1 tasks, percentage of best-performing kernels (%), and average cost per kernel (USD).

Method	MatMul	Activation Function	Convolution	Reduction/Normalization	Loss	Total
CuTeGen profiling from start	1.18 $\times$	1.80 $\times$	0.45 $\times$	2.08 $\times$	4.01 $\times$	1.34 $\times$
CuTeGen without profiling	1.47 $\times$	1.76 $\times$	0.57 $\times$	1.25 $\times$	4.04 $\times$	1.23 $\times$
CuTeGen with cuda	1.25 $\times$	1.79 $\times$	0.48 $\times$	1.05 $\times$	4.06 $\times$	1.12 $\times$
CuTeGen	1.62 $\times$	1.82 $\times$	0.61 $\times$	1.42 $\times$	4.06 $\times$	1.35 $\times$

Table 2: Average speedup over PyTorch on KernelBench Level-1 tasks for different CuTeGen variants, grouped by kernel category.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Initial synthesis prompt used to generate the first candidate kernel.

Tool	Speedup $\uparrow$	Low Precision $\downarrow$	Correct $\uparrow$	Fast1 $\uparrow$	Speedup On Fast1 $\uparrow$	Best $\uparrow$
CudaForge	0.61 $\times$	5%	66%	16%	1.23 $\times$	2%
CuTeGen	1.32 $\times$	0%	94%	50%	1.65 $\times$	49%

Appendix

Table 3: Comparison of CuTeGen and CudaForge on KernelBench Level-1 workloads. We report average speedup, precision lowering rate (%), correctness (%), Fast1 (%), speedup on Fast1 tasks, percentage of best-performing kernels (%).

Tool	Speedup $\uparrow$	Low Precision $\downarrow$	Correct $\uparrow$	Fast1 $\uparrow$	Speedup On Fast1 $\uparrow$	Best $\uparrow$
CudaForge	1.19 $\times$	33%	59%	32%	1.67 $\times$	27%
CuTeGen	2.12 $\times$	0%	76%	22%	2.71 $\times$	21%

Appendix

Table 4: Comparison of CuTeGen and CudaForge on KernelBench Level-2 workloads. We report average speedup, precision lowering rate (%), correctness (%), Fast1 (%), speedup on Fast1 tasks, percentage of best-performing kernels (%).

Method	MatMul	Activation Function	Convolution	Reduction/Normalization	Loss	Total
CuTeGen profiling from start	16%	100%	18%	50%	100%	41%
CuTeGen without profiling	21%	93%	15%	66%	100%	43%
CuTeGen with cuda	26%	100%	18%	53%	100%	42%
CuTeGen	26%	100%	21%	77%	100%	50%

Appendix

Table 5: Fast1 over PyTorch on KernelBench Level-1 tasks for different CuTeGen variants, grouped by kernel category.

Method	MatMul	Activation Function	Convolution	Reduction/Normalization	Loss	Total
CuTeGen profiling from start	84%	100%	89%	73%	100%	85%
CuTeGen without profiling	89%	100%	89%	85%	100%	91%
CuTeGen with cuda	100%	100%	85%	73%	100%	88%
CuTeGen	100%	100%	92%	85%	100%	94%

Appendix

Table 6: Correctness percentage over PyTorch on KernelBench Level-1 tasks for different CuTeGen variants, grouped by kernel category.

GPU Kernel Generation

GPU (graphics processing unit) kernel generation automates writing and optimizing device code for faster computation across accelerators.

See all 67 papers

CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation
Feb 27, 2026Weinan Dai, Hanlin Wu, Qiying Yu +13GPU Kernel Generation GPU Kernel Optimization
ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization
Jun 24, 2026Jiading Gai, Shuai Zhang, Kaj Bostrom +6GPU Kernel Optimization High-Performance Computing
Amazon · Siemens · University of Minnesota
KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation
Sep 27, 2026Changxin Ke, Rui Zhang, Zixiang Fang +8GPU Kernel Generation LLM-Based Program Synthesis
State Key Lab of Processors, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Intelligent Software Research Center, Institute of Software, CAS +1

Feb 27, 2026J/K move · Enter open · S save

CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation

Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao+12

ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed

GPU Kernel Generation GPU Kernel Optimization RL for Code Generation Agentic RL

Open paper AskPDF arXiv

GPU kernel optimization is fundamental to modern deep learning but remains a highly specialized task requiring deep hardware expertise. Despite strong performance in general programming, large language models (LLMs) remain uncompetitive with compiler-based systems such as torch.compile for CUDA kernel generation. Existing CUDA code generation approaches either rely on training-free refinement or fine-tune models within fixed multi-turn execution-feedback loops, but both paradigms fail to fundamentally improve the model's intrinsic CUDA optimization ability, resulting in limited performance gains. We present CUDA Agent, a large-scale agentic reinforcement learning system that develops CUDA kernel expertise through three components: a scalable data synthesis pipeline, a skill-augmented CUDA development environment with automated verification and profiling to provide reliable reward signals, and reinforcement learning algorithmic techniques enabling stable training. CUDA Agent achieves state-of-the-art results on KernelBench, delivering 100%, 100%, and 92% faster rate over torch.compile on KernelBench Level-1, Level-2, and Level-3 splits, outperforming the strongest proprietary models such as Claude Opus 4.5 and Gemini 3 Pro by about 40% on the hardest Level-3 setting.

Abstract

GPU Kernel Generation

Explore similar work

CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation

Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation