Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Zhen Cheng; Hao-Bo Yang; Wan-Yi Huang; Jin-Long Li

cs.CLApr 7, 2026

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Authors: Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang, Jin-Long Li

Organizations: China Merchants Bank Artificial Intelligence Laboratory

Abstract

Key-Value (KV) cache memory and bandwidth increasingly dominate large language model inference cost in long-context and long-generation regimes. Architectures such as multi-head latent attention (MLA) and hybrid sliding-window attention (SWA) can alleviate this bound, but integrating them into existing models remains difficult. Prior methods impose fine-grained structural requirements on both source and target attention modules, which cannot meet the feasible requirement in practical deployment. We present Attention Editing, a practical framework for converting already-trained large language models (LLMs) with new attention architectures without re-pretraining from scratch. Attention editing replaces the original attention with a learnable target module and trains it using progressive distillation, consisting of (1) layer-wise teacher-forced optimization with intermediate activation supervision to prevent cold-start error accumulation, and (2) model-level distillation on next-token distributions, optionally regularized by weak feature matching. We instantiate the framework on two different target--MLA and GateSWA, a gated hybrid SWA design, and apply it to Qwen3-8B and Qwen3-30B-A3B. The resulting models maintain competitive performance while delivering substantial efficiency improvements, demonstrating that large-scale attention conversion is both feasible and robust. Notably, experiments are conducted on an Ascend 910B clusters, offering a practical training case study on domestic hardware.

Figures & tables

Figure 2: Illustration of the forward propogation in two stages of progressive distillation. (a) Block-wise teacher forcing distillation: the inputs to each layer comes from the original model; (b) Model-level distillation: each architecture maintain its own inputs. The gray shading indicates that these two modules can be merged into a single model and trained efficiently using sharding strategies such as FSDP or DeepSpeed.

Attention Mechanisms	KV cache per token	KV memory
Attention Mechanisms	KV cache per token	Qwen3-8B	Qwen3-30B-A3B
GQA	$2n_{g}d_{h}l$	$100\%$	$100\%$
MLA	$(d_{c}+d_{h}^{R})l$	28%	56%
GateSWA	$\approx 2n_{g}d_{h}\cdot\frac{l}{6}$	17%	17%

Table 1: Comparison (%) of KV cache per token among GQA, MLA, and GateSWA. We follow the notations in DeepSeek-V2 ( DeepSeek-AI, 2024a ) . Here the approximation ignores the bounded local-window (

w=128

) cache in SWA layers and only accounts for the unbounded cache carried by full-attention layers. Under a 5:1 sliding-to-full schedule, one out of every six layers is a full-attention layer.

Data source	Percentage
General domain	40%
Math & code	35%
Chinese corpora	25%

Table 2: Data mix in stage I.

Models	ARC-E	ARC-C	C-Eval	MMLU
Qwen3-8B	96.46	91.46	77.04	74.47
Qwen3-8B-GateSWA	96.54	90.35	76.82	73.26
Qwen3-8B-MLA	96.71	90.01	73.70	71.77
Qwen3-30B-A3B	96.89	92.58	83.06	78.68
Qwen3-30B-A3B-GateSWA	98.19	91.98	81.20	77.24
Qwen3-30B-A3B-MLA	98.15	93.77	81.05	76.99

Table 3: Performance (%) of different attention variants under few-shot evaluation. The model is required to directly output the answer, and accuracy is computed by matching the predicted answer against the ground truth.

Models	GSM8K (thinking)	C-Eval (thinking)
Qwen3-8B	95.98	83.80
Qwen3-8B-GateSWA	94.31	78.90
Qwen3-8B-MLA	95.00	79.20
Qwen3-30B	96.44	86.40
Qwen3-30B-A3B-GateSWA	95.15	81.28
Qwen3-30B-A3B-MLA	94.39	81.13

Table 4: Accuracy (%) of different attention-variant models on GSM8K and C-Eval in thinking-mode.

Figure 4: Performance under different concurrency levels for three input lengths. The upper row shows the variation in throughput , while the lower row shows the variation in TTFT. The experimental setup is as follows: inference is conducted on a single H800 GPU using vLLM version 0.16.0, with gpu-memory-utilization set to 0.8. Missing data in the figure indicate that the model is unable to handle that level of concurrency simultaneously.

Figure 5: Throughput under different concurrency levels. The experimental setup is as follows: inference is conducted on a single Ascend 910B using vLLM/vLLM-Ascend version 0.16.0. Missing data in the figure indicate that the model is unable to handle that level of concurrency simultaneously. The results about Qwen3-8B-MLA are missing, because the MLA kernels in higher-version vLLM-Ascend impose customized requirements on specific parameter configuration.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

Jun 30, 2026

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.

Wenhao Li, Jinhao Dong, Hailin Zhang +3

School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

KV Caching Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention KV-Cache Quantization

Ask PDF

Sep 17, 2026

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative

Δ\mathrm{NLL}=+0.01209

, while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final

Δ\mathrm{NLL}=+0.02073

. Integrated Memory keeps perplexity within

-0.07\%

+0.93\%

while reducing total cache by up to

6.01\%

. A sampled 200-item downstream sanity check gives

28.5\%

32.0\%

overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya

Institute for Smart System Technologies University of Klagenfurt Klagenfurt, Austria · University of Klagenfurt / Inst. f. Smart Systems Technologies, Austria & Faculte Polytechnique Universite de Kinshasa, DR-Congo

Language Model Post-Training Recurrent Transformers Memory-Augmented Language Models

Ask PDF

Apr 15, 2026

ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depend jointly on keys and values, most existing KV management methods rely on key-only pruning, since incorporating values incurs prohibitive overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. Rather than replacing KV selection, ART dynamically terminates redundant KV traversal on top of existing dense or sparse attention policies. We introduce a stability-based criterion that monitors both magnitude and directional changes of intermediate attention outputs and provideds a theoretical characterization of the resulting truncation error. Experiments on the LongBench and RULER Needle-in-a-Haystack tasks show that ART increases the generation throughput of existing KV-cache methods by up to 20%, without compromising the result quality.

Chen Qiu, Guozhong Li, Cristian McGee +2

King Abdullah University of Science and Technology · University of Central Florida

Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Management Efficient Attention

Ask PDF

Abstract

LLM Inference Acceleration

Explore similar work

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

ART: Attention Run-time Termination for Efficient Large Language Model Decoding