Hardware-Aware Features for CUTLASS Kernel Selection

Shriram Chandran; Dominic Rinderer; Yakup Budanaz; Alexandru Calotoiu; Marcin Copik; Torsten Hoefler

cs.LGSep 28, 2026

Hardware-Aware Features for CUTLASS Kernel Selection

Authors: Shriram Chandran, Dominic Rinderer, Yakup Budanaz, Alexandru Calotoiu, Marcin Copik, Torsten Hoefler

Organizations: ETH Zurich Zurich, Switzerland

Abstract

GPU libraries such as CUTLASS expose tens of thousands of semantically equivalent kernels for a single operation, making exhaustive autotuning expensive and execution-free selection difficult. Existing analytical selectors require hand-designed performance rules, while learned selectors operate on raw configuration parameters and must infer hardware consequences from data. We introduce a hardware-aware representation for CUTLASS kernel selection that augments candidate configurations with statically computable estimates of induced hardware behavior. We construct a dataset of 4.9 million CUTLASS kernels and train gradient-boosted and neural learning-to-rank models to rank candidates within each problem. On held-out exhaustive evaluation problems, hardware-aware representations reduce selection regret by up to 40% relative to structural baselines and 64.2% relative to NVIDIA's matrix-multiply heuristics. We further evaluate data-efficient cross-precision and epilogue-fusion transfer within CUTLASS GEMM, showing that explicitly representing candidate-induced hardware behavior provides a useful inductive bias for learned kernel selection.

Figures & tables

Figure 1 . H100 Programming Model

Knob	Values / domain	Meaning
tile_m	$\{64,128,192,256\}$	Thread-block $M$ tile extent
tile_n	$\{16,32,48,64,80,96,128,192,256\}$	Thread-block $N$ tile extent
tile_k	$\{32,64,128,256,512\}$	Thread-block $K$ tile / pipeline atom
stages	$\{2,\ldots,12\}$	Mainloop pipeline stage count
cluster_m , cluster_n	$\{1,2,4,8,16\}$ , $C_{M}C_{N}\leq 16$	CTA cluster shape in $M$ and $N$
cluster_k	fixed $1$	$K$ -clustering not searched

Table 1 . Configuration knobs exposed by the BF16 SM90 generator ( generate_search_space ).

Constraint	Reason
Stream-K $\Rightarrow$ cooperative mainloop	CUTLASS Stream-K scheduler requires the cooperative TMA mainloop.
Cooperative mainloop $\Rightarrow T_{M}\equiv 0\pmod{128}$	Two consumer warpgroups split the $M$ tile; each half must stay 64-row WGMMA aligned.
$S_{\mathrm{mainloop}}+S_{\mathrm{epi}}+2\,\mathrm{KiB}\leq 227\,\mathrm{KiB}$	Hopper shared-memory capacity per CTA; $S_{\mathrm{epi}}$ depends on epilogue schedule and tile via estimate_epilogue_smem_bytes .
If $A$ is column-major: $T_{K}\leq 256\,C_{N}$	TMA smem box $K$ extent is multicast-split across the $N$ cluster dimension.
If $B$ is row-major: $T_{K}\leq 256\,C_{M}$	Symmetric $K$ -box limit along the $M$ cluster dimension for the transposed operand.

Table 2 . Static validity constraints for BF16 SM90 GEMM configurations.

Figure 2 . Overview of our execution-free CUTLASS kernel selector, which maps each problem and candidate catalogue to hardware-aware features and ranks kernels without benchmarking.

Metric	Global $\rho$	Mean $\rho_{g}$	Median $\rho_{g}$
Throughput and pipeline activity
tensor_active_pct	$+0.992$	$+0.392$	$+0.492$
dram_throughput_pct	$+0.817$	$+0.408$	$+0.500$
compute_throughput_pct	$+0.977$	$+0.235$	$+0.382$
tma_active_pct	$+0.792$	$+0.181$	$+0.289$
sm_active_pct	$+0.910$	$-0.001$	$+0.024$

Table 3 . Spearman correlation between profiled execution metrics and measured benchmark throughput.

Feature	Global $\rho$	Mean $\rho_{g}$	Median $\rho_{g}$
tile_m	$+0.626$	$-0.261$	$-0.409$
tile_n	$+0.827$	$+0.080$	$+0.091$
tile_k	$-0.415$	$+0.257$	$+0.272$
k_iters	$+0.826$	$-0.260$	$-0.289$
reg_pressure_proxy	$+0.827$	$-0.222$	$-0.319$
cluster_size	$-0.337$	$-0.247$	$-0.223$

Table 4 . Correlation between selected static candidate quantities and measured throughput.

Regime	Mean $\rho_{g}$	Median $\rho_{g}$
Small	$+0.217$	$+0.262$
Medium	$+0.307$	$+0.412$
Large	$+0.245$	$+0.273$
Compute-heavy	$+0.265$	$+0.289$
Memory-heavy	$+0.238$	$+0.262$
Wide	$+0.549$	$+0.577$

Table 5 . Within-group correlation between

T_{K}

and throughput under selected problem regimes.

Group	Features and purpose
Structural core	$\log_{2}M,\log_{2}N,\log_{2}K$ ; operand layouts; input, accumulation, and output types; tile and instruction shapes; pipeline stages; mainloop and epilogue schedules; cluster shape; tile scheduler; and architecture identifiers.
Work decomposition	Output-tile counts, reduction iterations, edge-tile waste, SM subscription, final-wave efficiency, Stream-K applicability, and cluster fit.
Memory behavior	Problem and tile arithmetic intensity, operand re-streaming, working-set and resident-panel size relative to L2, and input, data/epilogue/conversion traffic.
Resource pressure	Bytes per pipeline stage, total mainloop and epilogue shared-memory storage, register-pressure proxy, and shared-memory- and register-limited occupancy.
Pipeline behavior	Estimated TMA load time, WGMMA compute time, producer–consumer ratio, pipeline-fill fraction, stage amortization, and interactions between tile depth, stage count, and occupancy.

Table 6 . Feature groups.

Threshold	Mean fraction	Median fraction	Minimum fraction
Within 1%	0.43%	0.17%	0.17%
Within 5%	2.41%	1.00%	0.33%
Within 10%	10.91%	4.42%	1.17%

Table 7 . Fraction of valid kernels close to the exhaustive oracle.

Figure 3 . Sampling regret versus candidate measurement budget on the exhaustive BF16 evaluation groups.

Figure 4 . Probability that the sampled candidate pool contains a near-optimal kernel.

Heuristic-biased fraction $\alpha$	Mean sampling regret	Groups within 5% of oracle
0.00	8.5%	25.6%
0.25	1.4%	87.0%
0.50	1.1%	88.9%
0.75	0.9%	91.3%
1.00	0.6%	97.1%

Table 8 . Ablation of the exploitation fraction at

B=2000

Shape			Kernel count
$M$	$N$	$K$	TN	TT	NN	NT	Total
2048	2048	2048	60,795	60,740	60,740	60,696	242,971
4096	4096	4096	60,795	60,740	60,740	60,696	242,971
64	64	64	32,190	32,135	32,135	32,091	128,551
128	128	128	60,795	60,740	60,740	60,696	242,971
256	256	256	60,795	60,740	60,740	60,696	242,971

Table 9 . Held-out evaluation shapes and valid kernel counts per layout.

Knob	Range / values	Notes
$\eta$ (learning rate)	$[5\times 10^{-3},\,0.3]$ log-uniform
max_depth	$\{4,\ldots,12\}$
min_child_weight	$[1,\,30]$ log-uniform
subsample	$[0.5,\,1.0]$	row subsampling
colsample_bytree	$[0.5,\,1.0]$	column subsampling
$\lambda$ (L2)	$[0.01,\,20]$ log-uniform

Table 10 . Hyperparameter search space for Model A (gradient-boosted trees).

Knob	Range / values	Notes
loss	$\{\mathrm{LambdaRank},\,\mathrm{RankNet},$ $\mathrm{mse}\}$	optional categorical
learning rate	$[10^{-4},\,5\times 10^{-3}]$ log-uniform	Adam
hidden topology	$10$ preset tuples	depths $2$ – $4$ , widths $32$ – $1024$
dropout	$[0,\,0.4]$	after each ReLU
weight decay	$\{0,10^{-7},10^{-6},10^{-5},10^{-4},$ $10^{-3}\}$
epochs	$\{20,\ldots,150\}$

Table 11 . Hyperparameter search space for Model B (MLP).

Figure 5 . MLP validation NDCG@10.

Figure 6 . XGBoost validation NDCG@10.

Figure 7 . Full and structural features across training objectives.

Figure 8 . Mean selection regret.

Figure 9 . CDF of per-group selection regret.

Method	Mean	Median	Within 1%	Within 5%	Top-1
MLP (full)	6.2%	2.6%	36.8%	63.2%	27.9%
MLP (structural)	7.4%	3.1%	32.4%	55.9%	26.5%
XGBoost (full)	6.4%	4.9%	25.0%	52.9%	19.1%
XGBoost (structural)	10.7%	8.3%	17.6%	39.7%	13.2%
nvMMH (best-of- $K$ )	12.0%	10.2%	0.0%	25.0%	0.0%
nvMMH (top-1)	17.3%	14.6%	0.0%	10.3%	0.0%

Table 12 . Selection quality on 68 exhaustively measured shape–layout groups.

Figure 10 . Fraction of evaluation groups selected within

5\%

of oracle throughput.

Figure 11 . Mean regret by GEMM shape regime.

Figure 12 . Regret and within-

5\%

rate by regime for the best full and structural variants.

Figure 13 . MLP of different parameter counts.

Figure 14 . XGB of different depths.

Panel of Figure 15 — Figure 15 . Model Generalization and Transfer experiments.

Model	Features	Geo-mean % roof	vs. nvMMH	Win %
MLP MSE	Full	11.9%	1.14 $\times$	79.6%
MLP MSE	Structural only	11.5%	1.10 $\times$	73.9%
XGBoost MSE	Full	11.8%	1.13 $\times$	77.4%
XGBoost MSE	Structural only	11.4%	1.09 $\times$	70.1%
nvMMH	—	37.6	10.5%	—	—

Table 13 . Broad GEMM evaluation on

8{,}000

held-out problems.

Dtype	Method	Coverage	Speedup	Win %
FP32	MLP (full)	89.5%	1.24 $\times$	74.6%
FP32	XGBoost (full)	96.7%	1.24 $\times$	78.4%
FP32	MLP (structural)	96.1%	1.18 $\times$	71.8%
FP32	XGBoost (structural)	91.7%	1.20 $\times$	68.6%
FP8 E4M3	MLP (full)	65.4%	1.19 $\times$	47.4%
FP8 E4M3	XGBoost (full)	73.0%	1.17 $\times$	51.2%

Table 14 . Cross-precision transfer at

100\%

of the target TN training shapes.

Figure 16 . Speedup vs. nvMMH.

Model	Features	vs. nvMMH	Win %
MLP MSE	Full	1.12 $\times$	96.0%
MLP MSE	Structural only	1.13 $\times$	92.7%
XGBoost MSE	Full	1.12 $\times$	84.8%
XGBoost MSE	Structural only	1.05 $\times$	59.6%
Ridge	Full	0.83 $\times$	40.4%
Ridge	Structural only	0.27 $\times$	0.0%

Table 15 . DeepBench throughput generalization on

151

problems. Speedup is geometric-mean throughput relative to nvMMH.

Split	$n$	MLP (full)	MLP (struct.)	XGBoost (full)
Training	107	1.09 $\times$	1.09 $\times$	1.07 $\times$
Inference server	39	1.25 $\times$	1.28 $\times$	1.29 $\times$
Inference device	5	1.08 $\times$	1.00 $\times$	1.04 $\times$

Table 16 . DeepBench speedup vs. nvMMH by catalogue split.

Explore similar work

Jun 15, 2026cs.LG

daVinci-kernel: Co-Evolving Skill Selection, Summarization, and Utilization via RL for GPU Kernel Optimization

GPU kernel optimization represents a paradigm where functional correctness is assumed and execution efficiency is the objective. We present daVinci-kernel, a reinforcement learning framework that couples skill discovery with skill exploitation through a dynamically evolving skill library. daVinci-kernel jointly trains three agents sharing one LLM backbone: a Skill Selection Agent that retrieves relevant techniques via BM25 and LLM reranking, a Policy Agent that generates multi-turn CUDA/Triton kernels conditioned on selected skills, and a Skill Summary Agent that distills successful rollouts into reusable skills. Candidate skills are added only after execution-based verification confirms reproducible speedups. All three agents share a single LLM backbone, are initialized via a structured SFT cold start on diversity-filtered data, and are then jointly optimized end-to-end with multi-turn REINFORCE and per-agent advantage estimation. On KernelBench, daVinci-kernel-14B achieves 37.2%, 70.6%, and 32.2% on Level 1, Level 2, and Level 3 under the Fast

_1

threshold, outperforming the strongest prior RL-trained model, Dr. Kernel-14B.

Dayuan Fu, Mohan Jiang, Tongyu Wang +5

1SII · 2SJTU · 3GAIR

Graphics Processing Unit Kernels Self-Evolving Skill Libraries Resource-Efficient Summarization

Ask PDF

Sep 24, 2026cs.DC

KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimization

Deep learning inference and training performance depends critically on GPU kernel efficiency. Modern compilers such as PyTorch Inductor automatically generate GPU kernels from high-level model code, but frequently underperform expert-written implementations by wide margins. Recent LLM-assisted kernel optimizers can close this gap for standalone kernels, yet treat compiled models as black boxes, generally optimizing individual standalone kernels without respecting the compiler's structural decisions or verifying the model end-to-end. We present KernelOPT, a multi-agent system that treats compiled models as structured artifacts. It preserves vendor library calls (cuBLAS, cuDNN) and exclusively targets generated Triton sub-kernels using five profiling-guided LLM agents. A four-gate verification cascade of static validation, multi-seed correctness, model-level float64-fallback verification, and performance gating filters candidates during optimization and verifies the re-stitched model end-to-end. If no candidate passes all four gates, the system preserves the compiler baseline. The system accepts PyTorch nn.Modules, standalone Triton kernels, and Helion kernels. Evaluated on 250 KernelBench problems, KernelOPT achieves geometric mean speedups over \texttt{torch.compile} of 1.40

\times

(Level 1: 51/100), 1.15

\times

(Level 2: 31/100), and 1.07

\times

(Level 3: 12/50) across all problems.

Aheli Poddar, Sanskar Prasad, Arindam Samanta +3

Red Hat

Graphics Processing Unit Kernels Pytorch Code Optimization Optimization Modeling High-Level Source Code Agentic Search

Ask PDF

Dec 2, 2025cs.LG

CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning

In this paper, we propose CUDA-L2, a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. Using CUDA execution speed as the RL reward, CUDA-L2 automatically optimizes HGEMM kernels across 1,000 configurations. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used torch.matmul to state-of-the-art Nvidia's closed-source libraries, i.e., cuBLAS, cuBLASLt. In offline mode, where kernels are executed consecutively without time intervals, CUDA-L2 yields +22.0% over torch.matmul on average; +19.2% over cuBLAS using the optimal layout configuration (normal-normal NN and transposed-normal TN); +16.8% over cuBLASLt-heuristic, which queries cuBLASLt library and selects the algorithm based on the heuristic's suggestion; and +11.4% over the most competitive cuBLASLt-AutoTuning model, which selects the fastest algorithm from up to 100 candidates from cuBLASLt's suggestions. In server mode, where kernels are executed at random intervals simulating real-time inference, the speedups further increase to +28.7%, +26.0%, +22.4%, and +15.9% for torch.matmul, cuBLAS, cuBLASLt-heuristic, and cuBLASLt-AutoTuning respectively. CUDA-L2 shows that even the most performance-critical, heavily-optimized kernels like HGEMM can be improved through LLM-guided RL automation by systematically exploring configuration spaces at scales impractical for humans. Project and code can be found at github.com/deepreinforce-ai/CUDA-L2

Songqiao Su, Xiaoya Li, Albert Wang +3

DeepReinforce Team

Cuda Matrix Multiplication

Ask PDF