AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization

Jiehao Wu; Zixiao Huang; Wenhao Li; Chuyun Shen; Junjie Sheng; Xiangfeng Wang

cs.LGMar 24, 2026

AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization

Authors: Jiehao Wu, Zixiao Huang, Wenhao Li, Chuyun Shen, Junjie Sheng, Xiangfeng Wang

Organizations: School of Computer Science and Technology, East China Normal University · School of Computer Science and Technology, Tongji University · Shanghai University of International Business and Economics · Key Lab of Mathematics and Engineering Applications (MoE), East China Normal University ·

Abstract

Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implementation: a host-side tiling program that controls data movement and a kernel program that schedules and pipelines computation. We present AscendOptimizer, an episodic agent that builds missing optimization knowledge from execution itself. For kernel optimization, AscendOptimizer rewinds strong implementations by removing optimizations in a controlled way, then keeps the changes whose removal measurably hurts performance as reusable experience for later rewriting. For host-side optimization, it runs profiling-in-the-loop evolutionary search to find valid, fast tiling and data-movement configurations directly from hardware feedback. This combination lets the agent improve kernel structure and host-side scheduling together. On a benchmark of 101 real AscendC operators, AscendOptimizer achieves a 1.21x geometric-mean speedup over the open-source baseline, and 53.47% of operators run faster than their references. Given a same budget of evaluations per operator, AscendOptimizer consistently outperforms Best-of-N sampling and OpenEvolve in terms of geometric mean speedup, fast_p tail speedup ratios, and overall optimization progress across varying budgets.

Figures & tables

Model	CUDA (Pass@1)	AscendC (Pass@1)
DeepSeek-R1	$52.6\text{\,}\mathrm{\%}$	$1.4\text{\,}\mathrm{\%}$
Claude-Sonnet-4	$47.0\text{\,}\mathrm{\%}$	$2.1\text{\,}\mathrm{\%}$
Qwen3-235B (think)	$44.2\text{\,}\mathrm{\%}$	$0.7\text{\,}\mathrm{\%}$

Table 1: One-shot operator generation pass rate (Pass@1) across hardware platforms, reported by MultiKernelBench [ 37 ] . The results confirm severe knowledge scarcity on Ascend.

Figure 1: Overview of AscendOptimizer . Offline, we rewind benchmark operators and distill validated optimization motifs into a shared experience bank. Online, the bank guides Stage I kernel refinement, after which Stage II searches host-side tiling configurations under the refined kernel.

Panel of Table 2 — Table 2: Main performance. GM is geometric-mean speedup over cann-ops ; fast p is the fraction of operators exceeding $p\times$ speedup.

Figure 2: Optimization progress under hardware-evaluation budget. AscendOptimizer remains ahead of Best-of- $N$ sampling and OpenEvolve across the evaluated budgets.

Variant	Enabled components			Performance
	Rewrite	Experience	Tiling	GM $\uparrow$	$>1.2$ $\uparrow$	$>1.4$ $\uparrow$	$>1.8$ $\uparrow$	$>2.0$ $\uparrow$
Stage I only	✓	✓	✗	1.16	18.81%	12.87%	9.90%	7.92%
Stage I w/o Experience	✓	✗	✗	1.09	12.62%	8.74%	5.83%	3.88%
Stage II only	✗	✗	✓	1.02	2.97%	1.98%	0.00%	0.00%
AscendOptimizer	✓	✓	✓	1.21	18.81%	12.87%	10.89%	9.90%

Table 3: Component ablation. Results are computed over all 101 operators. Each variant is run independently with the same 230-evaluation budget per operator. Columns

>p

report fast p , the fraction of operators with speedup greater than

p\times

. Higher is better.

Figure 3: Cross-operator transfer for upsample_nearest_exact3d . A loop-consolidation pattern mined from clip_by_value_v2 transfers to the target operator; without retrieval, oracle bottleneck guidance rarely finds the same rewrite.

Panel of Figure 4 — Figure 4: Code-and-curve case study for matmul_all_reduce , showing the different roles of kernel refinement and host-side tiling search.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Aspect	Official Ascend C Best Practices	Experience Bank in This Work
Knowledge form	Principle-level guidance for human developers	Concrete optimization episodes with code-level rewrites and applicability conditions
Primary role	Explain common optimization principles	Support direct retrieval and transfer during automated operator optimization
Covered and aligned with docs	Tiling, DMA efficiency, double buffering, vectorized reduction	Recovers and operationalizes these documented principles in reusable optimization records
Beyond documented scope	Usually does not enumerate long-tail operator-specific rewrites	Includes loop restructuring, scalar index simplification, early-exit/search pruning, fine-grained synchronization tuning, and specialized SIMD paths
Granularity of reuse	Broad recommendations	Mechanism- and rewrite-level transferable patterns

Appendix

Table 4: How the experience bank differs from and extends official Ascend C best practices

GPU Kernel Optimization

GPU kernel optimization improves the speed and memory efficiency of computations by tailoring code to GPU hardware and workloads.

See all 209 papers

CANNBot-CAKE: Collaborative Agentic Kernel Evolution for Ascend NPU
Oct 22, 2026Deng Pan, Xue-Jian Gao, Junyi Fan +8Agentic Code Generation Neural Network Hardware Acceleration
Huawei
AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution
Jul 29, 2026Junhao Qiu, Zidong Wang, Yansong Sun +3AI Accelerator Inference Large Language Model-Guided Optimization
Department of Computer Science, City University of Hong Kong · Hong Kong Yuwin Technology Co., Ltd
Beyond Experience: Knowledge-Guided Evolution for NPU Kernel Generation
Oct 20, 2026Xi Chen, Ming Shen, Haotian Yan +2LLMs for Evolutionary Optimization
Shenzhen Loop Area Institute (SLAI) · Guangzhou University · Shanghai Jiao Tong University +2

Oct 22, 2026J/K move · Enter open · S save

CANNBot-CAKE: Collaborative Agentic Kernel Evolution for Ascend NPU

Deng Pan, Xue-Jian Gao, Junyi Fan, Bin Du+7

Huawei

Agentic Code Generation Neural Network Hardware Acceleration

Open paper AskPDF arXiv

Efficient custom operators are essential for modern deep learning workloads, yet developing high-performance kernels for Ascend NPUs remains slow, expert-intensive, and difficult to automate. Although AI agents now approach expert performance on GPU kernel benchmarks, current models remain less capable of generating AscendC kernels for NPUs, potentially reflecting the relative scarcity of relevant training data. We thus present CAKE, a CANN-native system for deployable custom operators that combines adaptive multi-DSL routing, separate Generation and Evolution modes, and reviewed knowledge reuse through CAKE Book and a Self-Evolving Agent. On 53 CANN-Bench v1.1 operators on Ascend 950PR, CAKE's aggregated results achieve

100\%

case pass rate on the standard set (1,060/1,060) and

98.28\%

on the hidden set (4,167/4,240), with an overall mean score of 78.63 out of 100 and an average performance speedup of

1.334\times

over the baseline. Evolution then pushes correct operators further: EngramGateFusion cuts aggregate device time by

46.6\%

over 20 rounds and roughly halves the mean roofline gap. These results offer a practical path toward closing Ascend's AI-for-kernel capability gap.

Abstract

GPU Kernel Optimization

Explore similar work

CANNBot-CAKE: Collaborative Agentic Kernel Evolution for Ascend NPU

AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution

Beyond Experience: Knowledge-Guided Evolution for NPU Kernel Generation