Key-Value (KV) cache memory and bandwidth increasingly dominate large language model inference cost in long-context and long-generation regimes. Architectures such as multi-head latent attention (MLA) and hybrid sliding-window attention (SWA) can alleviate this bound, but integrating them into existing models remains difficult. Prior methods impose fine-grained structural requirements on both source and target attention modules, which cannot meet the feasible requirement in practical deployment. We present Attention Editing, a practical framework for converting already-trained large language models (LLMs) with new attention architectures without re-pretraining from scratch. Attention editing replaces the original attention with a learnable target module and trains it using progressive distillation, consisting of (1) layer-wise teacher-forced optimization with intermediate activation supervision to prevent cold-start error accumulation, and (2) model-level distillation on next-token distributions, optionally regularized by weak feature matching. We instantiate the framework on two different target--MLA and GateSWA, a gated hybrid SWA design, and apply it to Qwen3-8B and Qwen3-30B-A3B. The resulting models maintain competitive performance while delivering substantial efficiency improvements, demonstrating that large-scale attention conversion is both feasible and robust. Notably, experiments are conducted on an Ascend 910B clusters, offering a practical training case study on domestic hardware.
Figures & tables
Figure 2: Illustration of the forward propogation in two stages of progressive distillation. (a) Block-wise teacher forcing distillation: the inputs to each layer comes from the original model; (b) Model-level distillation: each architecture maintain its own inputs. The gray shading indicates that these two modules can be merged into a single model and trained efficiently using sharding strategies such as FSDP or DeepSpeed.
Attention Mechanisms
KV cache per token
KV memory
Qwen3-8B
Qwen3-30B-A3B
GQA
2ngdhl
100%
100%
MLA
(dc+dhR)l
28%
56%
GateSWA
≈2ngdh⋅6l
17%
17%
Table 1: Comparison (%) of KV cache per token among GQA, MLA, and GateSWA. We follow the notations in DeepSeek-V2 ( DeepSeek-AI, 2024a ) . Here the approximation ignores the bounded local-window ( w=128 ) cache in SWA layers and only accounts for the unbounded cache carried by full-attention layers. Under a 5:1 sliding-to-full schedule, one out of every six layers is a full-attention layer.
Data source
Percentage
General domain
40%
Math & code
35%
Chinese corpora
25%
Table 2: Data mix in stage I.
Models
ARC-E
ARC-C
C-Eval
MMLU
Qwen3-8B
96.46
91.46
77.04
74.47
Qwen3-8B-GateSWA
96.54
90.35
76.82
73.26
Qwen3-8B-MLA
96.71
90.01
73.70
71.77
Qwen3-30B-A3B
96.89
92.58
83.06
78.68
Qwen3-30B-A3B-GateSWA
98.19
91.98
81.20
77.24
Qwen3-30B-A3B-MLA
98.15
93.77
81.05
76.99
Table 3: Performance (%) of different attention variants under few-shot evaluation. The model is required to directly output the answer, and accuracy is computed by matching the predicted answer against the ground truth.
Models
GSM8K (thinking)
C-Eval (thinking)
Qwen3-8B
95.98
83.80
Qwen3-8B-GateSWA
94.31
78.90
Qwen3-8B-MLA
95.00
79.20
Qwen3-30B
96.44
86.40
Qwen3-30B-A3B-GateSWA
95.15
81.28
Qwen3-30B-A3B-MLA
94.39
81.13
Table 4: Accuracy (%) of different attention-variant models on GSM8K and C-Eval in thinking-mode.
Figure 4: Performance under different concurrency levels for three input lengths. The upper row shows the variation in throughput , while the lower row shows the variation in TTFT. The experimental setup is as follows: inference is conducted on a single H800 GPU using vLLM version 0.16.0, with gpu-memory-utilization set to 0.8. Missing data in the figure indicate that the model is unable to handle that level of concurrency simultaneously.
Figure 5: Throughput under different concurrency levels. The experimental setup is as follows: inference is conducted on a single Ascend 910B using vLLM/vLLM-Ascend version 0.16.0. Missing data in the figure indicate that the model is unable to handle that level of concurrency simultaneously. The results about Qwen3-8B-MLA are missing, because the MLA kernels in higher-version vLLM-Ascend impose customized requirements on specific parameter configuration.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Institute for Smart System Technologies University of Klagenfurt Klagenfurt, Austria · University of Klagenfurt / Inst. f. Smart Systems Technologies, Austria & Faculte Polytechnique Universite de Kinshasa, DR-Congo