Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis

Yujie Zheng; Zhuo Li; Shengtao Zhang; Hanjing Wang; Junjie Sheng; Jiaqian Wang; Junchi Yan; Weinan Zhang

cs.LGMar 11, 2026

Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis

Authors: Yujie Zheng, Zhuo Li, Shengtao Zhang, Hanjing Wang, Junjie Sheng, Jiaqian Wang, Junchi Yan, Weinan Zhang,

Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Independent Researcher · MemTensor (Shanghai) Technology Co., Ltd

Abstract

Deploying Large Language Models to data-scarce programming domains poses significant challenges, particularly for kernel synthesis on emerging Domain-Specific Architectures where a "Data Wall" limits available training data. While models excel on data-rich platforms like CUDA, they suffer catastrophic performance drops on data-scarce ecosystems such as NPU programming. To overcome this cold-start barrier without expensive fine-tuning, we introduce EvoKernel, a self-evolving agentic framework that automates the lifecycle of kernel synthesis from initial drafting to continual refining. EvoKernel addresses this by formulating the synthesis process as a memory-based reinforcement learning task. Through a novel value-driven retrieval mechanism, it learns stage-specific Q-values that prioritize experiences based on their contribution to the current objective, whether bootstrapping a feasible draft or iteratively refining latency. Furthermore, by enabling cross-task memory sharing, the agent generalizes insights from simple to complex operators. By building an NPU variant of KernelBench and evaluating on it, EvoKernel improves frontier models' correctness from 11.0% to 83.0% and achieves a median speedup of 3.60x over initial drafts through iterative refinement. This demonstrates that value-guided experience accumulation allows general-purpose models to master the kernel synthesis task on niche hardware ecosystems. Our official page is available at https://evokernel.zhuo.li.

Figures & tables

Model	Level	CUDA (%)	Ascend C (%)
GPT-5.2	L1	92.0	14.0
GPT-5.2	L2	90.0	2.0
DeepSeek-V3.2	L1	50.0	8.0
DeepSeek-V3.2	L2	9.0	0.0
Qwen3-Coder-30B	L1	46.0	7.0
Qwen3-Coder-30B	L2	10.0	0.0

Table 1: Few-shot functional correctness (pass@4) of frontier LLMs on CUDA vs. Ascend C kernel generation. Results are from our experiments; the level definitions (L1, L2) and setup details are consistent with Section 4.1 .

Figure 1: The EvoKernel framework. (Left) Cold-Start Drafting: Given task batch $\mathcal{X}$ , retrieves top- $k$ candidates, filters context via $Q$ , and synthesizes an initial kernel. (Center) Environment & Memory: A multi-gate verifier assesses generated code to yield rewards, which update $Q$ via value iteration; code and results are stored in Memory. (Right) Continual Refining: Exploits generation traces $\mathcal{P}(x)$ and historical attempts, including observable child nodes, to iteratively optimize for lower latency.

Model	Method	Level 1		Level 2		Overall
Model	Method	CR (%)	Acc (%)	CR (%)	Acc (%)	CR (%)	Acc (%)
Qwen3-Coder-30B	Pass@ $k$	(22.0) 30.0	(7.0) 8.0	(0.0) 2.0	(0.0) 0.0	(11.0) 16.0	(3.5) 4.0
	Refinement	(13.0) 22.0	(2.0) 6.0	(0.0) 1.0	(0.0) 0.0	(6.5) 11.5	(1.0) 3.0
	Ours	(25.0) 33.0	(6.0) 11.0	(1.0) 3.0	(0.0) 0.0	(13.0) 18.0	(3.0) 5.5
DeepSeek-V3.2	Pass@ $k$	(21.0) 33.0	(7.0) 9.0	(1.0) 13.0	(0.0) 0.0	(11.0) 23.0	(3.5) 4.5
	Refinement	(16.0) 44.0	(0.0) 12.0	(2.0) 26.0	(0.0) 0.0	(9.0) 35.0	(0.0) 6.0

Table 2: Compilation Rate (CR) and Correctness (Acc) across difficulty levels, shown as (Round 1) Final , respectively representing the start point and the final performance. Notably, the huge gap between GPT-5.2 and other models implies that frontier LLMs with stronger in-context learning capability benefit substantially more from experience-driven methods.

Figure 2: Optimization outcomes. (Left) Category-level correctness and speedup distribution at budget

T{=}30

; color segments show the fraction of correct kernels in each speedup tier relative to Torch-NPU. (Right) Within-operator speedup achieved by iterative refinement across 159 operators with

\geq

1 valid optimization candidate beyond the initial correct draft; inset panels detail representative optimization trajectories.

Figure 3: Transfer and generalization. (Left) Transfer across difficulty levels: cumulative success rate on L2 under different stream compositions. (Right) Transfer across generator backbones: performance on held-out operators when reusing memory built with GPT-5.2.

Setup	CR (%)	Acc (%)
L2 Scratch	88.0	34.0
L1+L2 Mixed	98.0	53.0
L1 $\rightarrow$ L2	97.0	64.0

Table 3: Cross-level transfer summary on L2 at final iteration.

Workload	Platform	# Ops	CR (%)	Acc (%)	Fast 1 Ratio (%)
Attention Set	CUDA	70	100.0	97.1	72.1
Attention Set	Ascend	70	100.0	78.6	21.8
KernelBench	CUDA	250	100.0	100.0	68.0
mHC Kernels (DeepSeek)	Ascend	15	86.7	66.7	60.0

Table 4: Initial scaling-out results beyond the main KernelBench study.

Figure 4: mHC Kernels (Ascend): Optimization timeline and performance vs. Torch-NPU baseline for 15 DeepSeek mHC operators over 30 iterations (merged across three experiment series). (Left) Correctness and performance optimization timeline. (Right) Best correct run vs. baseline in log 2 speedup.

Figure 5: Retrieval ablations. (Left) Value-driven vs. heuristic retrieval on L2 operators (same L1 memory and

\epsilon

-greedy schedule). (Right) Effect of increasing retrieval pool size

K

at iteration 24; cumulative correctness and compilation rates on L1 operators.

Figure 6: Experience transfer dependency graph of 36_RMSNorm_ . Arrows trace causal references at first-solve iterations, revealing an emergent curriculum from simple to complex operators.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Operator	Torch-NPU ms	EvoKernel ms	Speedup
32_HardTanh	23.873	4.199	5.69 $\times$
30_Softsign	34.756	9.598	3.62 $\times$
45_Average_Pooling_2D	3814.723	1725.443	2.21 $\times$
20_LeakyReLU	9.525	9.511	1.00 $\times$

Appendix

Table 5: Operators where EvoKernel outperforms Torch-NPU in latency.

Figure 7: Attention Set (CUDA): Optimization timeline and performance vs. PyTorch baseline for 70 attention operators over 30 iterations. (Left) Correctness and performance optimization timeline. Green dots denote first-solve events; blue dots denote significant (

>

10%) performance improvements; orange dots denote the best performance achieved. (Right) Best correct run vs. baseline in log 2 speedup.

Figure 8: Codex (GPT-5.2) cumulative correctness.

Aspect	Refinement	Codex
Execution model	API conversation loop	Autonomous tool use
Iteration control	External script	Agent decides
Prompt updates	Each turn rebuilt	Single prompt
History length	1–2 turns	Internal memory
Feedback source	Injected by script	Agent calls validator
File operations	Extract from text	apply_patch tool

Appendix

Table 6: Comparison of Iterative Refinement and Codex agent architectures.

Level 1 Operators (30 total)
Operator	Type	Operator	Type
43_Max_Pooling_3D	pooling	46_Average_Pooling_3D	pooling
42_Max_Pooling_2D	pooling	49_Max_reduction_over_a_dimension	pooling
92_cumsum_exclusive	loss	93_masked_cumsum	loss
10_3D_tensor_matrix_multiplication	matmul	77_conv_transposed_3D_square_input_square_kernel	convolution
66_conv_standard_3D_asym_input_asym_kernel	convolution	38_L1Norm_	normalization

Appendix

Table 7: Randomly sampled operator subset for transfer evaluation.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 443 papers

From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis
Sep 28, 2026Longxiao Fan, Tao Zhang, Han Yan +5LLM-Based Program Synthesis Agent Skill Learning
Fudan University · University of Science and Technology of China · Huawei Technologies Ltd.
Beyond Experience: Knowledge-Guided Evolution for NPU Kernel Generation
Oct 20, 2026Xi Chen, Ming Shen, Haotian Yan +2LLMs for Evolutionary Optimization
Shenzhen Loop Area Institute (SLAI) · Guangzhou University · Shanghai Jiao Tong University +2
Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation
Jul 2, 2026Junyi Wen, Ruiyan Zhuang, Yongjia Xu +7Code Generation LLM-Based Program Synthesis
Sun Yat-sen University, Zhuhai, China · Greater Bay Area National Technology Innovation Center, Guangzhou, China · Peng Cheng Laboratory, Shenzhen, China

Sep 28, 2026J/K move · Enter open · S save

From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis

Longxiao Fan, Tao Zhang, Han Yan, Jiajun Li+4

Fudan University · University of Science and Technology of China · Huawei Technologies Ltd.

LLM-Based Program Synthesis Agent Skill Learning Memory-Augmented Agents

Open paper AskPDF arXiv

High-performance kernels underpin efficient accelerator execution but require expert tuning and lengthy manual optimization cycles. LLM coding agents promise automation, yet their CUDA knowledge transfers poorly to data-scarce domain-specific architectures (DSAs) such as NPUs, whose execution models and memory hierarchies differ substantially from those of GPUs. To address this transfer gap, post-training methods adapt LLMs to NPU programming but depend on scarce expert data and substantial training compute. Memory-learning agents instead adapt through external memory, but their uniform credit assignment gives adopted and unused experiences the same reward target, potentially biasing subsequent retrieval rankings. Moreover, when learned values guide only retrieval, high-value experiences that generalize across operators must be retrieved repeatedly rather than retained in context, thereby increasing retrieval overhead and weakening cross-task guidance. We therefore present SAGE, a persistent self-improving agent for NPU kernel synthesis. Adoption-Traced Utility estimation (ATU) combines explicit adoption records with kernel evaluation outcomes for adoption-aware credit assignment. Utility-Gated Consolidation (UGC) uses positive utility and repeated adoption across operators to select and abstract reusable rules into a bounded resident context. On NPUKernelBench, SAGE achieves a 95.5% execution rate versus 84.1% for the strongest controlled baseline, with 86.9% of solved operators outperforming torch_npu. With GLM-5.3, SAGE achieves a 43.99x speedup over the torch_npu reference on sparse flash attention. These results show that adoption-aware credit assignment and selective consolidation enable agents to accumulate and reuse hardware-specific knowledge across tasks.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis

Beyond Experience: Knowledge-Guided Evolution for NPU Kernel Generation

Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation