A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs

Chen Zhang; Yan Ding; Haotian Wang; Chubo Liu; Keqin Li; Kenli Li

cs.DCApr 10, 2026

A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs

Authors: Chen Zhang, Yan Ding, Haotian Wang, Chubo Liu, Keqin Li, Kenli Li

Organizations: Hunan University Changsha, China

Abstract

During the deployment of Large Language Models (LLMs), the autoregressive decoding phase on heterogeneous NPU platforms (e.g., Ascend 910B) faces severe memory-bound challenges. This study reveals the ``Model Scaling Paradox'' caused by the static deployment of single-sized models. It also points out the kernel synchronization overhead of fine-grained speculative decoding \cite{leviathan2023fast, chen2023speculative} under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD)

Figures & tables

Figure 1. The overall system architecture of the A-IO framework. The diagram illustrates the macro-orchestration flow, including the dual-model FP16 concurrent residency, 1B probe-based intent sensing, and the dynamic bandwidth-aware routing mechanism.

Model	2K Context Acc (%)	32K Context Acc (%)
1B Baseline	67.68	66.66
7B Baseline	62.80	95.73

Table 1. Accuracy Comparison under 2K and 32K Context Lengths (Human-eval Benchmark)

True \ Pred	Code	QA	Math	Recall
Code	94	4	2	94.0%
QA	8	89	3	89.0%
Math	1	6	93	93.0%

Table 2. 1B Probe Intent Classification Confusion Matrix

Configuration	C-eval		MMLU		GSM8K		Human-eval		QGPA
Configuration	Acc (%)	TPS	Acc (%)	TPS	Acc (%)	TPS	Acc (%)	TPS	Acc (%)	TPS
Static Baselines:
1B Baseline	63.20	21.58	71.17	21.87	73.92	21.44	67.68	21.18	39.90	20.09
1B PLD	64.40	26.54	65.29	27.08	62.09	26.64	51.22	27.63	33.33	27.35
1B Quant	57.20	21.20	62.74	21.50	71.80	21.10	57.32	20.90	40.40	19.80
7B Baseline	78.89	17.18	90.21	17.17	83.02	16.65	62.80	16.65	44.44	15.72

Table 3. End-to-End Performance (Accuracy and TPS) across 2K Context Benchmarks

Method	Scenario A		Scenario B		Scenario C
Method	Acc (%)	TPS	Acc (%)	TPS	Acc (%)	TPS
Static 1B Baseline	67.41%	21.28	67.76%	21.41	64.93%	14.50
Static 7B Baseline	68.04%	16.75	75.30%	16.86	87.31%	11.20
Random Routing	67.72%	19.01	71.53%	19.13	76.12%	12.85
A-IO (Actual)	70.85%	19.80	76.50%	18.15	87.32%	13.40

Table 4. Aggregate Accuracy and TPS under Diverse Mixed Workload Scenarios. (Note: The 13.40 TPS for A-IO in Scenario C is the weighted average performance based on the specific workload distribution. All data in Table 4 is strictly consistent with the per-task benchmarks in Table 3 .)

Ablation Configuration	Acc (%)	TPS
w/o Dynamic Model Routing (7B Only)	68.48	17.20
w/o Dynamic PLD Switch (PLD Off)	68.20	18.20
w/o Entropy Fallback (No validation)	65.10	20.10
Full A-IO (Actual)	70.85	19.80

Table 5. Ablation Study on A-IO Components (Scenario A)

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,055 papers

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative Decoding LLM Inference Acceleration
College of Computer Science, Chongqing University
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference Efficiency AI Accelerator Inference
Tsinghua University · Beijing Jiaotong University
First-Order LoRA Training with Phone NPUs
Oct 17, 2026Xu Zheng, Wei Cheng, Zhuomin Chen +4On-Device Training LLM Fine-Tuning
Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA +2

Mar 9, 2026J/K move · Enter open · S save

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

Chang Han, Yijie Hu, Jingling Liu

College of Computer Science, Chongqing University

Speculative Decoding LLM Inference Acceleration AI Accelerator Inference

Open paper AskPDF arXiv

Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.

Abstract

LLM Inference Acceleration

Explore similar work

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

First-Order LoRA Training with Phone NPUs