LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

Hui Zeng; Pengfei Yang; Yanxin Chen; Fusong Ju; Xinran Wei

cs.AIAug 22, 2026

LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

Authors: Hui Zeng, Pengfei Yang, Yanxin Chen, Fusong Ju, Xinran Wei

Organizations: Xidian University · Zhongguancun Academy · National Key Laboratory of Advanced Communication Networks, Shijiazhuang, Hebei, P. R. China · Zhongguancun Institute of Artificial Intelligence

Abstract

Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91 $\times$ /6.98 $\times$ geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745 $\times$ GeoMean speedup on KernelBench Level 2.

Figures & tables

Figure 1: Divergent isolated and deployment outcomes for an optimized ResNetConvLayer . The left panel contrasts standalone cold-cache benchmarking with in-model warm-context execution for the same candidate kernel. The right panel reports speedup from standalone kernel testing, the same kernel measured inside the model context, and the resulting full-model end-to-end execution.

Figure 2: Deployment-specific manifestations of the benchmark-to-deployment gap. The left panel shows how a latent out-of-bounds access can pass isolated qualification under a fresh allocator state yet fail after integration into the full model runtime. The right panel shows that an attention candidate specialized for prefill can be inapplicable to decode attention, so a kernel-level gain does not necessarily transfer across generation phases.

Figure 3: Overview of LLM4LLM. The framework starts from a target inference script, identifies deployable hotspots through real-model profiling, constructs phase-aware tasks, searches for candidate kernels with experience-guided agentic optimization, and accepts patches only after in-model correctness and latency validation. Validation feedback is returned to the search loop, while accepted patches produce the optimized deployment.

Figure 4: Experience-guided episodic optimization. Candidate generation and validation form a bounded episode. At the episode boundary, LLM4LLM summarizes the useful validation evidence into compact experience, archives the transient repair history, and restarts search from the task specification plus the distilled record.

Figure 5: Scope-matched comparison on transformer-family attention workloads. Each panel reports latency for eager execution, LLM4LLM, and expert deployment kernels across representative workload settings on A100 and H100. Lower is better.

Model	A100			H100
Model	Eager	Ours	Speedup	Eager	Ours	Speedup
Qwen3-4B	693.9	229.0	3.03 $\times$	380.2	100.7	3.78 $\times$
Qwen3-32B	3032.3	1387.1	2.19 $\times$	1516.7	594.4	2.55 $\times$
Llama-2-7B	658.8	353.9	1.86 $\times$	329.8	148.4	2.22 $\times$
Llama-2-13B	1082.2	603.7	1.79 $\times$	538.1	280.7	1.92 $\times$
Mamba (130M)	626.4	28.0	22.37 $\times$	537.7	16.4	32.79 $\times$

Table 1: End-to-end latency (ms) across language-model families on A100 and H100. Lower latency and higher speedup are better.

Figure 6: Scope-matched comparison on state-space mixer workloads. Each panel reports latency for eager execution, LLM4LLM, and Mamba-family fast paths based on mamba_ssm and causal-conv1d across representative workload settings on A100 and H100. Lower is better.

A100-SXM4-80GB							H100-80GB-HBM3
Method	Lang.	Pass	Fast1	Fast2	Mean	GeoMean	Method	Lang.	Pass	Fast1	Fast2	Mean	GeoMean
CUDA-L1	CUDA	100	98	–	3.55	–	CUDA-L1	CUDA	96	96	–	6.64	–
KernelSkill	CUDA	100	100	–	2.82	–	KernelBlaster	CUDA	81	84.85	–	10.223	2.592
STARK (A100-40G)	CUDA	100	100	–	2.69	–	QiMeng-Kernel	Triton	99	86	12	1.28	–
QiMeng-Kernel	Triton	99	66	8	1.22	–	AI CUDA Engineer	CUDA	82	61.33	–	1.356	1.214
LLM4LLM	Triton	100	97	39	22.157	2.745	LLM4LLM	Triton	99	95	41	15.847	2.628

Table 2: KernelBench Level-2 comparison on A100 and H100 GPUs. Pass, Fast1, and Fast2 are reported as percentages. Mean and GeoMean are speedups over PyTorch eager unless otherwise specified.

Model	Method	Pass	Fast1	Fast2	Mean	GeoMean	P50	P75	Fast1	Fast2	Mean	GeoMean	P50	P75
		vs. PyTorch eager							vs. torch.compile
GPT-5.4	Sample-1	46	4	1	0.693	0.555	0.511	0.640	2	1	0.643	0.507	0.500	0.562
	Sample-10	75	63	8	1.417	1.230	1.064	1.336	44	9	1.344	1.124	1.009	1.323
	Iter-10	100	64	4	1.171	0.999	1.011	1.220	41	5	1.065	0.887	0.980	1.144
	LLM4LLM w/o restart (10)	100	89	32	6.782	1.961	1.250	4.221	66	31	6.573	1.742	1.143	3.986
	LLM4LLM (10)	100	91	32	14.962	2.005	1.332	4.110	71	31	14.716	1.781	1.230	4.020

Table 3: KernelBench Level-2 ablation against PyTorch eager and torch.compile . Pass, Fast1, and Fast2 are reported as percentages; Mean, GeoMean, P50, and P75 are speedups.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Family	Module	Phase / bucket	Main specialization	Task speedup
Transformer	LlamaAttentionPrefill	q_len > 1	Causal online-softmax attention with fused QKV wiring	2.40 $\times$
Transformer	Qwen3AttentionDecode	q_len = 1	Grouped-query decode sharing each KV tile across query heads	2.01 $\times$
Mamba2	Mamba2MixerPrefill	prefill scan	Compact recurrent scan with cache-safe mixer wiring	14.31 $\times$
Mamba	MambaMixerDecode	single-token decode	Fused Conv1d–SiLU–projection chain for cached decode	2.06 $\times$
Recurrent	RecurrentGemmaRglru	recurrent update	Fused gate normalization plus recurrent-state scan	6.85 $\times$

Appendix

Table A.4: Representative accepted Triton kernels on extracted deployment tasks, measured on A100. Speedup is measured on the extracted validation task for the module, before full-model aggregation.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

FastKernels: Benchmarking GPU Kernel Generation in Production
May 22, 2026Gabriele Oliaro, Yichao Fu, May Jiang +5GPU Kernel Optimization GPU Acceleration
1Snowflake AI Research · 2CMU · 3UCSD +1
KernelSight-LM: A Kernel-Level LLM Inference Simulator
Jun 26, 2026Xiteng Yao, Taeho Kim, Hengzhi Pei +7LLM Inference LLM Inference Serving
Amazon Web Services Santa Clara, CA, USA · Boston University Boston, MA, USA
Harness Engineering for LLM-Driven GPU Kernel Generation
Jul 20, 2026Yue Shui, Chenyu Ma, Hangfei Xu +2GPU Kernel Optimization Large Language Model-Guided Optimization
Baidu, Inc.

May 22, 2026J/K move · Enter open · S save

FastKernels: Benchmarking GPU Kernel Generation in Production

Gabriele Oliaro, Yichao Fu, May Jiang, Owen Lu+4

1Snowflake AI Research · 2CMU · 3UCSD+1

GPU Kernel Optimization GPU Acceleration LLM Inference High-Performance Computing Large Language Model-Guided Optimization GPU Kernel Generation+1

Open paper AskPDF arXiv

LLM-based agents for GPU kernel generation are advancing rapidly, yet their progress is fundamentally constrained by the benchmarks they optimize against. Existing benchmarks are poorly aligned with production inference frameworks: they evaluate kernels on a single GPU with synthetic inputs, ignore the surrounding compilation stack, and reward replicating known optimizations rather than discovering new ones. The resulting reward signals are misleading: agents learn to generate kernels that score well in sandboxes but introduce interface incompatibilities, compilation-stack conflicts, and silent correctness degradation when integrated into real systems. We introduce FastKernels, a kernel benchmark built around a minimal set of 46 representative architectures spanning 8 categories, whose kernels collectively subsume those of 96.2% (409/425) of HuggingFace Transformers architectures. FastKernels doubles as a minimalistic, production-grade inference framework that runs at parity with hardened systems such as vLLM and SGLang on mainstream LLM serving and substantially exceeds upstream references on under-served architectures; each task's interface mirrors the corresponding module in the state-of-the-art library for its architecture family, enabling direct deployment of optimized kernels into production codebases. Evaluating state-of-the-art kernel agents on FastKernels, we find that even the strongest agent achieves only 0.94

\times

aggregate speedup over production baselines, with weaker agents at

0.78\times

and

0.53\times

-- confirming that benchmark-production misalignment is a critical bottleneck for the field. We release FastKernels as a stepping stone toward kernel agents whose benchmark gains translate directly into production throughput improvements. Code is available at https://github.com/Snowflake-AI-Research/fastkernels

Abstract

LLM Inference Acceleration

Explore similar work

FastKernels: Benchmarking GPU Kernel Generation in Production

KernelSight-LM: A Kernel-Level LLM Inference Simulator

Harness Engineering for LLM-Driven GPU Kernel Generation