As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models spanning language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike prior benchmarks that evaluate kernels primarily relative to software implementations, SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes. To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static analysis based checks against common reward-hacking strategies. SOL-ExecBench reframes GPU kernel benchmarking from beating a mutable software baseline to closing the remaining gap to hardware Speed-of-Light.
Figures & tables
Benchmark
Problems
Source
Metric
Precision
Target
Train
HW
KernelBench
270
Curated + models
fastp
FP32/FP16/BF16
Any
×
FlashInfer-Bench
26
Inference traces
fastp
FP16/BF16/FP8
BW +
×
BackendBench
271
PyTorch OpInfo
Speedup
FP32/FP16/BF16
Any
Partial
TritonBench
184
GitHub repos
Correctness+
FP32/FP16/BF16
Any
×
ComputeEval
232
Synthetic
pass@1
FP32/FP16
Any
×
Table 1: Comparison of GPU kernel generation benchmarks.
Figure 1: Overview of the SOL-ExecBench construction pipeline. Input is 124 source models spanning six domains; output is 235 validated benchmark problems in four categories.
Cat.
Description
#
Precision
Examples
L1
Single-operation kernels extracted from real models; building blocks of neural network computation
94
BF16 / FP32
GQA, RMSNorm, SwiGLU, RoPE
L2
Multi-operation fused kernels representing complete computational blocks; 3–10 × more complex than L1
Standalone inference primitives from three production model families (Llama-3.1-8B, Qwen3-30B-A3B, DeepSeek-V3/R1)
26
BF16 / FP8
Fused attention, FP8 MoE, RMSNorm
Total
235
Table 2: Benchmark category summary.
Figure 2: Problem characterization of the 235 SOL-ExecBench problems across four dimensions. (a) Benchmark category breakdown (forward/backward split shown with hatching). (b) Distribution by primary operation type. (c) Problems by model domain, stacked by benchmark category. (d) Distribution by primary compute precision.
Figure 3: SOLAR pipeline for deriving the Speed-of-Light bound TSOL from a PyTorch model and input shape.
Figure 4: SOLAR pipeline on a concrete SOL-ExecBench L1 problem.
Figure 5: SOL score as a function of kernel runtime Tk , shown for TSOL=50 and Tb=100 . The metric is anchored at S=1 when Tk=TSOL and at S=0.5 when Tk=Tb , and decays smoothly toward 0 as runtime increases. The curve also highlights the metric’s nonlinearity: the same runtime improvement yields a larger score gain when it occurs closer to the SOL regime.
Category
Exploit Description
Defense Mechanism
Concurrency
Thread Injection: Hiding work on unrecorded Python threads.
Thread count monitoring.
Stream Injection: Hiding work on unsynchronized CUDA streams.
Disabling multi-stream usage.
JIT Forking: Abusing torch.jit.fork for parallel execution.
LLM-judge static analysis.
State & Caching
Reuse Cached Output: Caching outputs based on data_ptr and returning them during timing.
Input cloning; custom memory allocator to prevent address caching; LLM-judge.
Lazy Evaluation: Returning FakeTensor s that only compute during eq validation.
Strict type checking ( type(t) is torch.Tensor ).
One-time Correctness: Skipping math after the first successful validation pass.
Multiple correctness trials with randomized inputs
Table 3: Observed reward hacking strategies by agentic kernel optimizers and corresponding mitigations.
Figure 9
Figure 8: Validating the SOL score against fraction of headroom reclaimed. Both panels share the same x-axis.
Figure 9: Distribution of reward hacking exploits detected across agent submissions by exploit type and problem category is shown here.
Figure 10: Agent solution (new scoring baseline) results are analyzed here.
Optimizing GPU kernels with LLM agents is an iterative process over a large design space. Every candidate must be generated, compiled, validated, and profiled, so fewer trials will save both runtime and cost. We make two key observations. First, the abstraction level that agents operate at is important. If it is too low, the LLM wastes reasoning on low-impact details. If it is too high, it may miss important optimization choices. Second, agents cannot easily tell when they reach the point of diminishing returns, wasting resources as they continue searching. These observations motivate two design principles to improve efficiency: (1) a compact domain-specific language (DSL) that can be learned in context and lets the model reason at a higher level while preserving important optimization levers, and (2) Speed-of-Light (SOL) guidance that uses first-principles performance bounds to steer and budget search. We implement these principles in μCUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels that covers kernel configuration, epilogue fusion, and multi-stage pipelines. We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the benchmark. On 59 KernelBench problems with the same iteration budgets, switching from generating low-level code to DSL code using GPT-5-mini turns a 0.40x geomean regression into a 1.27x speedup over PyTorch. Adding SOL-guided steering raises this to 1.56x. Across model tiers, μCUTLASS + SOL-guidance lets weaker models outperform stronger baseline agents at lower token cost. SOL-guided budgeting saves 19-43% of tokens while retaining at least 95% of geomean speedup, with the best policy reaching a 1.68x efficiency gain. Lastly, SOL analysis helps detect benchmark-gaming cases, where kernels may appear fast while failing to perform the intended computation.
Siva Kumar Sastry Hari, Vignesh Balaji, Sana Damani +2
We present KernelArc, a multi-agent framework for autonomous GPU kernel optimization across heterogeneous workloads. Strategy-specialized agents run in parallel and coordinate through conclusions-only shared memory, a deterministic benchmark guard, and read-only cross-agent state with plateau-triggered drafting. We evaluate KernelArc on NVIDIA H100 and B200 GPUs using category-representative SOL-ExecBench workloads. The resulting implementations span custom BF16 GEMM, static cuBLASLt Expert-API configuration tables, fused mixture-of-experts backward, shape-gated decoder-layer fusion, native NVFP4 grouped-query attention, and paged prefill attention. In the public SOL-ExecBench leaderboard snapshot recorded on August~20, 2026, KernelArc ranked first on every representative L1, L2, Quantization, and FlashInfer task evaluated. The trajectories support the paper's central motivation: shared multi-agent search can broaden exploration and reach stronger incumbents within a fixed candidate budget, while the value of individual coordination features depends on the kernel and optimization stage.
Joyjit Kundu, Ben Stoffelen, Kaili Wang +2
AILabs, Interuniversity Microelectronics Centre (IMEC)
GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial low-level expertise. Recent AI coding agents can iteratively read code, invoke compilers and profilers, and refine implementations, yet existing kernel benchmarks evaluate single LLM calls rather than full agent workflows, and none include both kernel-to-kernel optimization and unseen-configuration generalization testing. We present AgentKernelArena, an open-source benchmark for measuring AI coding agents on GPU kernel optimization. The benchmark contains 196 tasks spanning HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation, and evaluates complete agent workflows in isolated workspaces using gated compilation, correctness, and performance checks, centralized scoring and an unseen-configuration generalization protocol that tests whether optimizations transfer to input configurations the agent never observed. Across production agents including Cursor Agent, Claude Code, and Codex Agent, we find near-perfect compilation and high correctness rates on most task categories, with the strongest configurations achieving mean speedups of up to 6.89x on PyTorch-to-HIP, 6.69x on HIP-to-HIP, and 2.13x on Triton-to-Triton tasks. Our unseen-configuration evaluation shows that HIP-to-HIP and Triton-to-Triton optimizations largely transfer to unseen input shapes, while PyTorch-to-HIP exhibits substantial correctness drops, indicating that agents generating kernels from scratch frequently hardcode shape-specific assumptions. AgentKernelArena is designed as a modular, extensible framework for rigorous evaluation of agentic GPU kernel optimization across agents, tasks, and hardware targets.
Sharareh Younesian, Wenwen Ouyang, Sina Rafati +11