Ask, Don't Equip: Question-Driven Coding Agents for Deep Kernel Optimization
Organizations: Snowflake AI Research · Carnegie Mellon University
Abstract
LLM agents for GPU kernel optimization sit at two ends of a structure-freedom tradeoff. Scripted agents fix what the model does at each step, via a learned policy, an optimization menu, or fixed roles, and are sample-efficient where the recipe applies and stall where it does not. Open-ended agents let a self-directed model mutate a population of kernels and pay for the freedom in trials: thousands of evaluations, repeated edits, and wrong candidates. Both respond to hard targets by equipping the agent with more: parsed profiler output, disassembly analysis, retrieval over documentation and libraries, and stores of lessons. We argue that both lack direction, a hypothesis about where the remaining gap is, and that a short question asked at the right moment supplies it without narrowing what the agent may do next. We present KernelSeek, a question-driven agent that makes an existing CUDA kernel faster with no operator-specific knowledge and no faster implementation to imitate: a general-purpose coding agent, a minimal harness that enforces a verifiable goal, and questions at decision points, first from a human, then from a bank the agent applies to itself. On the same task, evaluator, hardware, and week, a tool-rich autonomous agent built in the prevailing style plateaued at 0.50x FlashAttention-3 after 133 model-driven steps; KernelSeek reached 1.13x. It then produced dependency-free FlashAttention forward kernels for Hopper and Blackwell, single CUDA files with no CUTLASS, CuTe, cuBLAS, or cuDNN, correct on all 182 scenarios per architecture and 1.16x FlashAttention-3 and 1.46x FlashAttention-4 (geometric mean). Across 162 human messages and 11.5K agent tool calls, every large gain followed a question that moved the agent to a different layer of the system or rejected a claimed limit, never one that supplied information. We release the kernels, harness, question bank, and all traces.
Figures & tables
| Set | Correct | Speedup vs. fastest … | |||
|---|---|---|---|---|---|
| upstream | open-source | available | |||
| Hopper (H200), upstream = FA3 | |||||
| attntest | 64 | 64/64 | 1.22 | 1.22 | 1.14 |
| edge | 31 | 31/31 | 1.55 | 1.55 | 1.31 |
| claim | 77 | 77/77 | 1.01 | 1.01 | 0.99 |
| real | 10 | 10/10 | 1.10 | 1.10 | 1.08 |
| Equipped agent | KernelSeek | |
| Human text in prompts | none | 40 messages, 17 questions |
| Model-driven steps / turns | 133 steps | 40 turns |
| Wall-clock to best | 44.7 h of 62.4 h | 68.0 h, 41.3 h active |
| API cost | $660 | not recoverable (Cursor) |
| Candidates correct | 132 / 133 | all reported |
| Best geomean | 0.48 | 1.13 |
| Question (category) | Agent’s belief | Measurement it triggered | Outcome |
|---|---|---|---|
| “What configs are bringing down the geomean?” (gap triage) | Tuning the hd128 paths was the right priority. | Per-configuration table: hd96, hd192, hd256 were on the scalar fallback at 0.01 . | Fallback removed; 0.44 0.96 geomean in 4 hours. |
| “Is this ceiling a property of the hardware, or of our design? Where does the time go outside the kernel?” (bound differencing) | Split-KV decode was at a “genuine ceiling within the harness rules”. | Host-side profile: a per-call tensor fill of 2–5 s outside the kernel. | Fill removed; split-KV 0.87 0.97, geomean 1.02 1.13. |
| “Do our scenarios cover the shapes and features the operator is actually called with?” (benchmark validity) | 40 of 40 scenarios correct meant the kernel was done. | The operator’s full test grid ported: 52 of 58 correct, with sink and left-pad outputs off by 495 and 103 . | Six fixes; 58 of 58 correct, geomean 1.10 on the new grid. |
| “Why do you think the goal was achieved? We are still well below parity.” (challenge a limit) | The Stop-hook judge had accepted “goal met”. | Agent: “I stopped after the first successful lever and declared victory prematurely.” | fp8 hd256 decode 0.20/0.34 0.66/0.77 after “this is not a rare corner”. |
| “Is it NCU that hangs, or our kernel?” (hypothesis check) | “No deadlock in our kernel.” | Sanitizer, then debugger: a warp-convergent tcgen05.wait inside a divergent branch. | All 8 hanging shapes fixed. |
| “Is this really a register floor?” (challenge a limit) | hd256 was register-capped; the judge had accepted the limit twice. | A collective rewrite of the consumer loop. | 0.84 0.95–0.96 ; the agent’s own note: “the ‘floor’ conclusion was wrong”. |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Examples | Documented outcome | |
|---|---|---|---|
| Measurement check | 13 | “are you sure these improvements are not just noise? you used the noise level from a previous run”; “what is the overall performance compared to FA3? Wasn’t it 0.96 before?”; “are the values in the tables latencies or speedups?” | Re-measurement on an idle GPU; two claimed wins withdrawn. |
| Bound differencing | 10 | “isn’t the original FA3 code constrained by the same harness rules as our implementation? Why can FA3 reach high performance and we can’t?”; “can you explain what you mean by ‘genuine ceilings within the harness rules’?” | Host-side fill found; split-KV 0.87 0.97, geomean 1.02 1.13. |
| Benchmark validity | 7 | “what shapes does FA3 use for testing?”; “are the scenarios comprehensive compared with SOL-ExecBench?”; “how do we do in non-GQA test cases?” | Suite 40 58 105 scenarios; two silent wrong outputs found. |
| Gap triage | 7 | “what configs are bringing down the geomean? which ones are lower than 0.8x the speed of FA3?”; “what do the configs that are less than 0.95x of FA3 have in common? Please group the slow configs by issue” | Scalar fallback exposed; 0.44 0.96. |
| Agent process | 6 | “didn’t you already try that before?”; “of these gaps, which ones have we already tried and why did they fail?”; “why are you establishing the baselines for 2h? this should only have taken a few minutes” | Earlier transcripts reviewed before the next rewrite. |
| Challenge a limit | 5 | “why do you think the goal was achieved? we’re still significantly below 1.0x”; “this is not a rare corner, don’t be lazy. please keep going until you reach 1.0x parity” | Premature “goal met” retracted; fp8 hd256 0.20/0.34 0.66/0.77. |
| Set | Scenario | FA3 ( s) | ours | cuDNN | FlashInfer | best |
|---|---|---|---|---|---|---|
| attntest | attn_dense_hd64_causal | 138.4 | 1.40 | 1.60 | — | 0.88 |
| attntest | attn_dense_hd96_causal | 168.7 | 1.07 | 1.49 | — | 0.72 |
| attntest | attn_dense_hd128_causal | 135.3 | 1.06 | 1.16 | — | 0.91 |
| attntest | attn_dense_hd192_causal | 171.2 | 0.97 | 1.12 | — | 0.87 |
| attntest | attn_dense_hd256_causal | 203.6 | 0.94 | 1.09 | — | 0.87 |
| attntest | attn_dense_hd128_noncausal | 178.7 | 1.05 | 1.18 | — | 0.89 |
| Set | Scenario | FA4 ( s) | ours | cuDNN | FlashInfer | best |
|---|---|---|---|---|---|---|
| attntest | attn_dense_hd64_causal | 102.2 | 1.49 | 1.40 | 1.05 | 1.06 |
| attntest | attn_dense_hd96_causal | 104.6 | 1.00 | 1.33 | — | 0.75 |
| attntest | attn_dense_hd128_causal | 110.4 | 1.38 | 1.39 | 1.02 | 0.99 |
| attntest | attn_dense_hd192_causal | — | UNSUP(ours, FA4, FlashInfer) | |||
| attntest | attn_dense_hd256_causal | — | UNSUP(ours) | |||
| attntest | attn_dense_hd128_noncausal | 123.7 | 1.26 | 1.51 | 1.05 | 0.83 |
| Component | Distilled output in the prompt |
|---|---|
| Lever selector | one of 16 optimizations, keyed to the NCU bottleneck label |
| NCU parser | bottleneck diagnosis for the worst configuration |
| SASS analyzer | instruction-mix deltas vs. the champion and a CUTLASS oracle |
| Doc retrieval | BM25 over the PTX ISA, CUDA guide, NCU and Nsight docs, CUTLASS docs, Hopper tuning guide |
| Reference libraries | hints extracted from DeepGEMM, ThunderKittens, CUTLASS (1.6 GB on disk) |
| Knowledge store | typed notes (kernel, profiling, algorithmic), 15 seeded |
GPU Kernel Optimization
GPU kernel optimization improves the speed and memory efficiency of computations by tailoring code to GPU hardware and workloads.
See all 238 papers