MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs

Zile Wang; Zhide Xie; Tao Ji

cs.LGOct 12, 2026

MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs

Authors: Zile Wang, Zhide Xie, Tao Ji

Organizations: Fudan University

Abstract

Multi-Head Latent Attention (MLA) compresses the KV cache into a latent representation, reducing inference memory and bandwidth. Existing MHA2MLA/TransMLA methods do not support QKNorm and require costly recovery on modern LLMs, especially MoE. We present MLAfication, a QKNorm-compatible, parameter-efficient extension of MHA2MLA. It combines calibrated KNorm, parameter-efficient distillation, contribution-aware Partial-RoPE, and activation-aware joint KV initialization. Across MHA and GQA models, including QKNorm- and MoE-equipped variants, MLAfication preserves performance under substantial KV-cache compression, surpasses the current SOTA on commonsense and long-context benchmarks, and extends MLA conversion to modern architectures. On Qwen3-30B-A3B, MLAfication delivers up to $5.35\times$ the serving throughput of BF16 GQA and, combined with 4-bit KV quantization, reduces the KV-cache footprint by $94.73\%$ . Our code is available at https://github.com/JT-Ushio/MLAfication.

Figures & tables

Figure 1: Adapting QKNorm to efficient MLA decoding. For compactness in the schematic, we write $\bm{q}_{c}$ and $\bm{q}_{r}$ for $\bm{q}_{i,\mathrm{nope}}$ and $\bm{q}_{i,\mathrm{rope}}$ , respectively, and use the same convention for $\bm{k}$ .

	1K	2K	4K	8K	16K
$\alpha_{h}$	0.8243	0.8185	0.8228	0.8260	0.8305

Table 1: Stability of

\alpha_{h}

across context lengths, estimated on a calibration set of 64 samples.

Figure 2: Parameter-Efficient Distillation. Snowflakes and flames denote frozen and trainable modules, respectively.

Model	Tokens	KV Mem.	Avg@ $\to$	MMLU	ARC	PIQA	HS	OBQA	WG	Avg@LB
MHA foundation models
SmolLM-1.7B	1T	–	55.90	39.27	59.87	75.73	62.93	42.80	54.85	18.66
+ CPT	300M	–	55.42	38.68	58.79	76.12	61.33	42.80	54.78	16.76
+ TransMLA		-68.75%	55.24 ${}_{\text{-0.18}}$	38.60	58.95	74.97	61.52	43.00	54.38	15.33 ${}_{\text{-1.43}}$
+ MLAfication		-68.75%	55.03 ${}_{\text{-0.39}}$	38.14	58.72	75.63	60.52	42.60	54.54	15.59 ${}_{\text{-1.17}}$
+ CPT	700M	–	55.71	38.80	58.72	76.06	61.98	43.60	55.09	16.09

Table 2: Main results across MHA, GQA, QKNorm-based GQA, and QKNorm-based MoE models. Avg@ and Avg@LB denote the averages over short-context benchmarks and LongBench, respectively. Tokens denotes recovery-training tokens, and KV Mem. denotes the reduction from the original KV cache. Subscripts report absolute score differences from the CPT baseline.

Figure 3: Normalized inference throughput across prompt lengths and concurrency levels. Throughput is normalized to the baseline under the same setting; the dashed horizontal line denotes the baseline.

Model	KV Mem.	Avg@CS	Avg@LB
Qwen3-8B	–	64.15	30.96
w. KNorm removal	-43.75%	63.40	30.60
w. Ours	-43.75%	64.27	30.83
w. KNorm removal	-68.75%	62.16	30.09
w. Ours	-68.75%	62.98	31.06
w. KNorm removal	-81.25%	60.84	28.31

Table 3: Effect of QKNorm adaptation. Compared with directly removing KNorm, our method consistently improves downstream performance under different KV-cache reduction ratios.

Figure 4: KL-divergence loss curves for QKNorm ablation. Our method achieves lower loss and faster convergence than directly removing KNorm.

Precision	$\bm{r=/}$		$\bm{r=512}$		$\bm{r=256}$		$\bm{r=128}$
Precision	KV Mem.	Avg@LB	KV Mem.	Avg@LB	KV Mem.	Avg@LB	KV Mem.	Avg@LB
BF16	100.00%	32.65	-43.75%	31.36 -1.29	-68.75%	29.54 -3.11	-81.25%	28.01 -4.64
FP8 Quanto	-50.00%	31.97 -0.68	-71.88%	29.31 -3.34	-84.38%	28.98 -3.67	-90.63%	26.90 -5.75
Int4 HQQ	-71.88%	31.76 -0.89	-84.18%	29.19 -3.46	-91.21%	28.63 -4.02	-94.73%	27.00 -5.65
Int4 Quanto	-71.88%	31.47 -1.18	-84.18%	29.28 -3.37	-91.21%	28.89 -3.76	-94.73%	27.21 -5.44
Int2 HQQ	-84.38%	28.97 -3.68	-91.21%	25.11 -7.54	-95.12%	24.89 -7.76	-97.07%	21.70 -10.95

Table 4: LongBench results of Qwen3-30B-A3B-base under different latent dimensions and KV-cache precisions. KV-memory reductions are measured against the original BF16 cache; with a group size of 64, quantization results in effective precisions of 4.5 bits for Int4 and 2.5 bits for Int2. Subscripts denote absolute score differences from the original BF16 baseline (32.65).

$\mathcal{L}$	Fine-tuning	KV Mem.	Avg@CS	Avg@LB
SmolLM-1.7B-MLA
NTP	Full	-68.75%	54.69	14.53
NTP	QKV-only		54.51	13.72
Distill	Full		54.38	16.54
Distill	QKV-only		55.03	15.59
NTP	Full	-81.25%	53.43	13.55

Table 5: Ablation of distillation and QKV-only fine-tuning.

Low-rank KV Approximation
Stage	$\Delta\mathcal{M}$	KV Mem.	Attn. MSE $\downarrow$
MHA2MLA	w. SVD	-68.75%	482.43
MLAfication	w. SVDLLM V2	-68.75%	86.58
MHA2MLA	w. SVD	-81.25%	866.92
MLAfication	w. SVDLLM V2	-81.25%	244.69
MHA2MLA	w. SVD	-87.50%	1189.58

Table 6: Attention-editing ablation on Qwen2.5-7B, measured by the average layer-wise MSE between original and edited attention outputs before recovery.

Model	KV Mem.	Avg@CS	Avg@LB
LLaMA-2-7B	–	59.85	27.40
+ TransMLA	-68.75%	58.20	0 1.34
+ MLAfication	-68.75%	52.68	0 1.06
+ TransMLA	-87.50%	51.19	0 0.95
+ MLAfication	-87.50%	42.89	0 1.19
+ TransMLA	-92.97%	43.26	0 0.74

Table 7: Zero-shot TransMLA vs. MLAfication.

Method	KV Mem.	Max Vio.	Mean Vio.
Baseline	–	5.48	2.59
Ours	-43.75%	6.04	2.70
	-68.75%	5.74	2.69
	-81.25%	5.92	2.73
KNorm removal	-43.75%	10.53	4.21
	-68.75%	12.48	3.28

Table 8: MoE load-balancing violations on Qwen3-30B-A3B. Max Vio. and Mean Vio. denote the maximum and mean of the layer-wise maxima, respectively.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	KV Mem.	$\bm{d_{kv}}$	Tokens	LR	L
SmolLM-1.7B	-68.75%	1024	300M	$1e{-4}$	2K
SmolLM-1.7B	-81.25%	640	700M	$5e{-4}$	2K
SmolLM-1.7B	-87.50%	384	1B	$5e{-4}$	2K
Qwen2.5-7B	-43.75%	512	50M	$1e{-4}$	4K
Qwen2.5-7B	-68.75%	256	600M	$1e{-4}$	4K
Qwen2.5-7B	-81.25%	128	1.2B	$1e{-4}$	4K

Appendix

Table 9: Recovery hyperparameters for MLAfication. The trainable parameters are restricted to QKV projections and attention normalization parameters. Approximate trainable ratios are 17.6% for SmolLM-1.7B, 6.1% for Qwen2.5-7B, 11.1% for Qwen3-8B, and 1.6% for Qwen3-30B-A3B.

Qwen2.5-7B
Configuration	4K				8K				16K
	C32	C64	C128	C256	C32	C64	C128	C256	C32	C64	C128	C256
GQA, BF16	3942	5463	6762	7272	2963	3766	4301	3914	1988	2303	2083	2079
GQA, FP8	$1.09\times$	$1.14\times$	$1.16\times$	$1.20\times$	$1.17\times$	$1.20\times$	$1.22\times$	$1.41\times$	$1.23\times$	$1.26\times$	$1.50\times$	$1.39\times$
MLA-512, BF16	$1.03\times$	$1.13\times$	$1.21\times$	$1.26\times$	$1.15\times$	$1.24\times$	$1.32\times$	$1.44\times$	$1.27\times$	$1.35\times$	$1.55\times$	$1.38\times$
TransMLA-512, BF16	$1.01\times$	$1.10\times$	$1.19\times$	$1.23\times$	$1.13\times$	$1.22\times$	$1.31\times$	$1.41\times$	$1.26\times$	$1.34\times$	$1.55\times$	$1.37\times$

Appendix

Table 10: Absolute output-token throughput (tokens/s) for the GQA BF16 baselines and normalized speedups for all other configurations. Here, 4K, 8K, and 16K denote equal prompt and output lengths, and C denotes concurrency.

Qwen2.5-7B on Ascend 910B
Configuration	4K				8K				16K
	C16	C32	C64	C128	C16	C32	C64	C128	C16	C32	C64	C128
GQA, BF16	695.90	1040.95	1410.60	1417.94	546.54	736.61	765.13	788.98	382.66	396.36	412.04	426.99
MLA-512, BF16	$0.834\times$	$0.924\times$	$1.014\times$	$1.362\times$	$0.933\times$	$1.069\times$	$1.407\times$	$1.326\times$	$1.084\times$	$1.458\times$	$1.373\times$	$1.410\times$
MLA-128, BF16	$0.982\times$	$1.170\times$	$1.400\times$	$2.124\times$	$1.195\times$	$1.544\times$	$2.323\times$	$3.244\times$	$1.574\times$	$2.520\times$	$3.556\times$	$4.033\times$

Appendix

Table 11: Absolute output-token throughput (tokens/s) for the BF16 GQA baseline and normalized speedups for MLAfication on a single Ascend 910B NPU. Here, 4K, 8K, and 16K denote equal prompt and output lengths, and C denotes concurrency.

Panel of Table 12 — Table 12: Progressive search for KV-cache compression ratios on Qwen2.5-7B.

Explore similar work

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
Mar 18, 2026Zhongzhu Zhou, Fengxiang Bie, Ziyan Chen +6Multi-Head Latent Attention Low-Rank Approximation
University of Sydney · Together AI · University of Texas at Austin +1

Sep 29, 2026J/K move · Enter open · S save

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

Zunhai Su, Yuxuan Sun, Jianchao Tan, Tao Zhang+4

The University of Hong Kong · Meituan LongCat Team · South China University of Technology+1

LLM Inference Acceleration Attention Quantization KV-Cache Quantization Multi-Head Latent Attention Post-Training Quantization Low-Bit Quantization+1

Open paper AskPDF arXiv

Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. In this work, we establish a systematic model of MLA's dual-path quantization errors, characterizing their distinct effects on attention-output distortion and explaining the pronounced amplification of RoPE-path errors. Guided by this analysis, we introduce QuantMLA, a function-aligned framework for low-bit dual-path quantization. We derive path-specific transformation spaces that preserve full-precision computation while remaining fully fusible into model parameters offline, eliminating online transformation overhead. Within these spaces, QuantMLA learns path-specific transformations with function-aligned objectives: attention-output reconstruction captures the content path's coupled matching and aggregation errors, while positional QK reconstruction preserves the RoPE-induced component of the attention logits and admits a theoretical bound on output distortion. Across four MLA model families, QuantMLA enables, to our knowledge, the first reported joint INT4 caching of the content and RoPE caches with minimal accuracy degradation. Further compressing the content cache to INT2 while retaining the RoPE key cache at INT4 maintains competitive performance on challenging reasoning and code benchmarks. We develop a native low-bit MLA attention kernel that integrates unpacking and dequantization directly into attention computation. The physical cache layout provides 3.59x compression at 128K context, while a cache-pressure serving workload achieves 5.168x higher whole-job output throughput than BF16. The code will be released upon acceptance.

Abstract

Explore similar work

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

QK-Normed MLA: QK normalization without full key caching

CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention