Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization via an Experience-Driven Workflow and Experience Graph Memory

Siyuan Chen; Runlin Hou; Shenxiu Wu; Yansong Sun; Junming Cao; Yiyu Zhang; Shudi Shao; Junhao Qiu

cs.LGAug 26, 2026

Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization via an Experience-Driven Workflow and Experience Graph Memory

Authors: Siyuan Chen, Runlin Hou, Shenxiu Wu, Yansong Sun, Junming Cao, Yiyu Zhang, Shudi Shao, Junhao Qiu,

Organizations: Department of Computer Science, City University of Hong Kong · Application Software Engineering Lab, Huawei Technologies Ltd.

Abstract

Hardware kernel optimization requires repeated compilation, correctness testing, profiling, and revision. LLM agents can automate parts of this process, and stronger foundation models, longer context windows, and longer execution horizons have improved optimization within individual tasks. These advances alone do not enable an agent to learn from completed optimization runs. Existing kernel-optimization agents seldom preserve a decision, its observed execution feedback, and the later decisions that use that evidence. Retaining every prior trajectory is also impractical because an expanding history competes with the current task for context. We present KOPE, an experience-driven framework for hardware kernel optimization. KOPE records optimization trajectories with correctness and performance feedback in Experience Graph Memory, then uses Active Context Management and Injection to retrieve relevant experience under a fixed token budget. The graph retains decision order, observed outcomes, and alternative branches, allowing evidence collected on the target hardware to inform later optimization steps and tasks. Under the same GLM-5.2 setting, the geometric mean of KOPE's per-operator speedups is $1.54\times$ that of CANNBot, the strongest competing baseline. In a complete 53-operator ablation, Active Context Management and Injection raises pass rate from 60.0% to 84.6%, increases the evaluator-reported positive-field geometric mean from 0.0382 to 0.0661, and reduces optimization token consumption from 15.9B to 1.113B tokens relative to passive agent-led context construction. Enabling Experience Graph Memory raises full-suite pass rate from 55.2% to 84.6% and yields a $1.43\times$ geometric-mean speedup on valid timing comparisons. These results support continual optimization through external experience while the foundation model remains fixed.

Figures & tables

Figure 1: KOPE’s experience-driven workflow. With model parameters fixed, Active Context Management and Injection constructs a bounded prompt from required task state (hot), retrieved experience (warm), and broader knowledge (cold). Target-side execution returns compiler, correctness, and performance outcomes, which are recorded in a persistent exploration journal and Decision-to-Outcome Experience Graph. Retrieval informs the next decision within the current task and carries experience across tasks.

Model	System	Op. Pass	Case Pass	Pass Rate	Score
GLM-5.2	KOPE	52/53	897/1,060	84.6%	2004.49
	CANNBot	37/53	613/1,060	57.8%	1465.93
	CUDA-Agent	13/53	156/1,060	14.7%	312.00
Deepseek- V4-Pro	KOPE	47/53	783/1,060	73.9%	1242.90
	CANNBot	25/53	417/1,060	39.3%	1015.38
	CUDA-Agent	8/53	96/1,060	9.1%	192.00

Table 1: Best full-suite result for each model and system configuration. Op. Pass reports operators with at least one passing case over the 53-operator suite. Case Pass reports accepted cases over all 1,060 cases, with missing operators counted as failures.

Figure 2: Model and workflow comparison on CANN Bench. (a) Full-suite pass rate for KOPE, CANNBot, and CUDA-Agent over 53 operators and 1,060 cases. Cases from unreported operators count as failures. (b) Positive GLM-5.2 operator-speedup fields grouped by CANN Bench difficulty and workflow. Boxes show the interquartile range, black lines show the median, whiskers extend to

1.5\times

the interquartile range, and points are individual measurements. The logarithmic axis marks parity at

1\times

. KOPE, CANNBot, and CUDA-Agent contribute 52, 37, and 13 measurements, respectively. CUDA-Agent contributes 8/3/2/0 measurements from L1 to L4, all below parity. The plot omits missing and nonpositive fields and does not include Deepseek-V4-Pro speedups.

Context Policy	Case Pass	Pass Rate	Score	Speedup	Tokens
Passive	636/1,060	60.0%	636.00	0.0382 $\times$	15.9B
Active	897/1,060	84.6%	2004.49	0.0661 $\times$	1.113B
Effect	+261	+24.6 pp	+1368.49	$1.73\times$	$-93.0\%$

Table 2: Active versus passive context construction. Speedup reports the geometric mean over positive fields; Tokens reports aggregate use.

GraphMemory Configuration	Case Pass	Pass Rate	Speedup
GraphMemory Configuration	Case Pass	Pass Rate	L1	L2	L3	L4
With	897/1,060	84.6%	0.659 $\times$	0.0866 $\times$	0.0344 $\times$	0.0182 $\times$
Without	585/1,060	55.2%	0.664 $\times$	0.0595 $\times$	0.0229 $\times$	0.00199 $\times$
Effect	+312	+29.4 pp	$0.993\times$	$1.455\times$	$1.501\times$	$9.175\times$

Table 3: Experience Graph Memory ablation. Case Pass and Pass Rate use the full 1,060-case suite. With reports actual per-level geometric means of positive per-operator speedup fields; Without is derived using matched positive-timing ratios. Effect reports With relative to Without.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Measured outcomes for four KOPE knowledge settings on K3. Panel (a) reports pass coverage over the common 204-case universe. Panel (b) reports geometric-mean speedup over the fixed 123-case intersection passed by every setting, preventing coverage differences from changing the performance denominator. The dashed line marks PyTorch parity. Error bars are omitted because the archive contains per-case point estimates but not the underlying repeated timing samples.

Item	Value
Hardware	SpacemiT K3 (8 $\times$ X100, riscv64, RVV 1.0)
Memory	16 GB
System	Bianbu 4.0.4 (resolute)
Python	3.12.13
PyTorch	2.8.0+spacemit.0
Triton	3.6.0+spacemit.a5

Appendix

Table 4: RISC-V experimental environment.

Setting	Passed	Rate	GM pass	GM ∩
Cold start	123/204	60.3%	0.388 $\times$	0.388 $\times$
Ascend memory	170/204	83.3%	0.142 $\times$	0.145 $\times$
K3 memory	182/204	89.2%	0.508 $\times$	0.514 $\times$
Ascend + K3	188/204	92.2%	0.758 $\times$	0.786 $\times$

Appendix

Table 5: Measured KOPE outcomes on the K3 case universe.

Figure 4: Measured passed cases by operator and KOPE knowledge setting. Each cell shows passed/total cases; color encodes the corresponding pass rate. Rows are grouped as matmul, activation, elementwise, and reduction families. The archived pass sets are nested across the four settings; this is an observed property of these runs, not an assumption used by the analysis.

GPU Kernel Optimization

GPU kernel optimization improves the speed and memory efficiency of computations by tailoring code to GPU hardware and workloads.

See all 211 papers

KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts
May 27, 2026Shuoming Zhang, Qiuchu Yu, Ruiyuan Xu +11Agent Skill Learning GPU Acceleration
SKLP, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of New South Wales
Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization
May 19, 2026Dmitry Redko, Albert Fazlyev, Konstantin Sozykin +3GPU Kernel Optimization Large Language Model-Guided Optimization
Applied AI Institute · AI Talent Hub, ITMO University · YSDA +1
AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation
Apr 17, 2026Weihua Du, Jingming Zhuo, Yixin Dong +9Test-Time Adaptation GPU Kernel Optimization
1Carnegie Mellon University · University of Washington · 3Arm Ltd.

May 27, 2026J/K move · Enter open · S save

KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts

Shuoming Zhang, Qiuchu Yu, Ruiyuan Xu, Chenjing Zhang+10

SKLP, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of New South Wales

Agent Skill Learning GPU Acceleration High-Performance Computing Large Language Model-Guided Optimization Learning to Optimize GPU Kernel Generation+2

Open paper AskPDF arXiv

LLM-based agents are increasingly used to generate GPU kernels, but they often struggle to determine when an optimization is sound because its required code state and dependencies are implicit in expert implementations. We introduce KLineage, which learns this missing "when" knowledge from expert kernels: instead of relying on forward rollouts, KLineage walks expert implementations backward through validation-gated simplifications and reverses each accepted step into a reusable optimization skill. Each skill records not only the optimization intent, but also when to apply the optimization technique, including where it applies in code, what conditions made it valid, what effect it has, and what failures its assumptions avoid. A downstream LLM materializes these skills on new code surfaces under the same compile/correctness/profile gate. This guidance on when to apply each optimization can help downstream models to generate higher-performance kernels. On five expert workloads across two NVIDIA architectures, these lineage-derived skills serve as an effective optimization curriculum, exceeding recent memory-based LLM-kernel baselines in both final kernel quality and optimization efficiency under the same fixed budget. We also demonstrate that the KLineage framework extends beyond NVIDIA GPUs to Ascend NPUs. Our code is publicly available at https://github.com/ict-agent/klineage.

Abstract

GPU Kernel Optimization

Explore similar work

KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation