Multi-Head Low-Rank Attention

Songtao Liu; Hongwu Peng; Zhiwei Zhang; Zhengyu Chen; Yue Guo

cs.LGMar 2, 2026

Multi-Head Low-Rank Attention

Authors: Songtao Liu, Hongwu Peng, Zhiwei Zhang, Zhengyu Chen, Yue Guo

Organizations: The Pennsylvania State University · University of Connecticut · Carnegie Mellon University · University of California, Los Angeles

Abstract

Long-context inference in large language models is bottlenecked by Key--Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High-Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi-Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from a sharding bottleneck during distributed decoding via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced to redundantly load the complete KV cache for every token, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Extensive experiments show that MLRA achieves state-of-the-art perplexity and downstream task performance, while also delivering a 2.8 $\times$ decoding speedup over MLA. Code is available at https://github.com/SongtaoLiu0823/MLRA. Pretrained weights, along with the training and evaluation data, are available at https://huggingface.co/Soughing/MLRA.

Figures & tables

Method	# Parameters	KV Cache	Loading Per Token Per Device (1 GPU)	Loading Per Token Per Device (2 GPUs)	Loading Per Token Per Device (4 GPUs)	Loading Per Token Per Device (8 GPUs)
MHA ( Vaswani et al., 2017 )	$4dhd_{h}$	$2hd_{h}$	$128d_{h}$	$64d_{h}$	$32d_{h}$	$16d_{h}$
MQA ( Shazeer, 2019 )	$2dd_{h}\left(h+1\right)$	$2d_{h}$	$2d_{h}$	$2d_{h}$	$2d_{h}$	$2d_{h}$
GQA ( Ainslie et al., 2023 )	$2dd_{h}\left(h+g\right)$	$2gd_{h}$	$16d_{h}$	$8d_{h}$	$4d_{h}$	$2d_{h}$
MLA ( DeepSeek and others, 2024a )	$d_{c}^{\prime}\left(d+hd_{h}+hd_{h}^{R}\right)+dd_{h}^{R}$ $+\,d_{c}\left(d+2hd_{h}\right)+dhd_{h}$	$d_{c}+d_{h}^{R}$	$4.5d_{h}$	$4.5d_{h}$	$4.5d_{h}$	$4.5d_{h}$
MFA ( Hu et al., 2024 )	$d_{c}^{\prime}\left(d+h\cdot 2d_{h}\right)$ $+2d\cdot 2d_{h}+dh\cdot 2d_{h}$	$4d_{h}$	$4d_{h}$	$4d_{h}$	$4d_{h}$	$4d_{h}$
TPA ( Zhang et al., 2025 )	$d(\beta_{q}+2\beta_{kv})\left(h+d_{h}\right)+dhd_{h}$	$2\beta_{kv}\left(h+d_{h}\right)$	$6d_{h}$	$5d_{h}$	$4.5d_{h}$	$4.25d_{h}$

Table 1: Comparison of parameters and KV cache loading among attention mechanisms. Some results are taken from Zhang et al. (2025) and Zadouri et al. (2025) . For attention mechanism details, refer to Appendix C .

Method	MHA	MQA	GQA	MLA	MFA	TPA	GLA-2	GTA	MLRA-2	MLRA-4
Arithmetic Intensity	$\frac{4nhd_{h}}{4nhd_{h}}$	$\frac{4nhd_{h}}{4nd_{h}}$	$\frac{4nhd_{h}}{4ngd_{h}}$	$\frac{4nhd_{c}+2nhd_{h}^{R}}{2n\left(d_{c}+d_{h}^{R}\right)}$	$\frac{4nh\cdot 2d_{h}}{4n\cdot 2d_{h}}$	$\frac{4nh\beta_{kv}d_{h}+4nhd_{h}}{4n\beta_{kv}\left(h+d_{h}\right)}$	$\frac{2nh\frac{d_{c}}{2}+nhd_{h}^{R}}{2n\left(\frac{d_{c}}{2}+d_{h}^{R}\right)}$	$\frac{4nhd_{h}}{2n\left(gd_{h}+d_{h}^{R}\right)}$	$\frac{2nh\frac{d_{c}}{4}+nhd_{h}^{R}}{2n\left(\frac{d_{c}}{4}+d_{h}^{R}\right)}$	$\frac{4nh\frac{d_{c}}{4}+2nhd_{h}^{R}}{2n\left(\frac{d_{c}}{4}+d_{h}^{R}\right)}$
Arithmetic Intensity	$\approx 1$	$\approx h$	$\approx\frac{h}{g}$	$\approx 2h$	$\approx h$	$\approx\frac{\left(1+\beta_{kv}\right)hd_{h}}{\beta_{kv}\left(h+d_{h}\right)}$	$\approx h$	$\approx\frac{2h}{g}$	$\approx h$	$\approx 2h$

Table 2: Comparison of attention decoding arithmetic intensity among attention mechanisms.

Method	Wikipedia	C4	Pile	RefinedWeb	Cosmopedia	FineWeb	FineWeb-Edu	Avg
MHA	14.624	16.575	12.929	18.698	9.102	15.656	9.434	13.860
MQA	15.134	16.837	14.008	19.202	9.484	15.942	9.533	14.306
GQA	15.057	16.628	13.758	18.885	9.504	15.713	9.427	14.139
MLA	14.567	16.345	12.965	18.523	8.966	15.440	9.284	13.727
MFA	15.693	16.738	13.903	19.125	9.423	15.815	9.506	14.315
TPA	14.789	16.622	13.333	18.971	9.130	15.717	9.333	13.985

Table 3: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, RefinedWeb, Cosmopedia, FineWeb, and FineWeb-Edu. The best results are indicated in bold , while the second best are underlined .

Method	ARC-E	ARC-C	OpenBookQA	BoolQ	HellaSwag	Winogrande	PIQA	Avg
MHA	69.11	39.16	40.80	62.26	60.82	57.62	74.86	57.81
MQA	66.16	38.31	41.80	62.05	60.24	59.83	74.48	57.55
GQA	67.13	39.42	42.00	63.39	61.29	56.91	75.08	57.89
MLA	68.22	39.16	42.60	64.10	61.39	60.06	75.68	58.75
MFA	69.02	39.93	42.40	63.49	60.72	58.96	75.19	58.53
TPA	69.44	40.61	41.60	60.03	61.02	57.85	74.54	57.87

Table 4: Downstream evaluation on seven common-sense reasoning benchmarks: ARC-E, ARC-C, OpenBookQA, BoolQ, HellaSwag, Winogrande, and PIQA. ARC-E/C, OpenBookQA, HellaSwag, and PIQA use normalized accuracy (%); others use standard accuracy (%). Best is bold ; second best is underlined .

Method	Wikipedia	C4	Pile	RefinedWeb	Cosmopedia	FineWeb	FineWeb-Edu	Avg
GQA w/ gating	14.362	16.484	13.113	18.696	9.098	15.581	9.311	13.806
MLA w/ gating	14.346	16.297	12.866	18.456	8.936	15.383	9.212	13.642
GLA-2 w/ gating	14.597	16.286	12.997	18.473	8.986	15.369	9.198	13.701
MLRA-2 w/ gating	14.424	16.252	13.017	18.407	8.924	15.351	9.180	13.651
MLRA-4 w/ gating	14.431	16.170	13.073	18.386	8.874	15.266	9.148	13.621

Table 5: Validation perplexity (lower is better) w/ gating across seven datasets. The best results are indicated in bold , while the second best are underlined .

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Shape / Type	Meaning
$n$	scalar	Sequence length (number of tokens).
$d$	scalar	Model/hidden dimension.
$h$	scalar	Number of attention heads.
$d_{h}$	scalar	Per-head dimension.
$d_{h}^{R}$	scalar	Partial RoPE dimension.
$d_{r}$	scalar	RoPE rotation dimension in Theorem 1 (even); typically $d_{r}=d_{h}$ or $d_{r}=d_{h}^{R}$ .

Appendix

Table 6: Notation used throughout this paper.

Model Size	# Parameters	# Layers	$h$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	24	3072	128	8192

Appendix

Table 7: Model configuration of MHA for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.00M	24	24	1	3072	128	10152

Appendix

Table 8: Model configuration of MQA for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	24	6	3072	128	9728

Appendix

Table 9: Model configuration of GQA for main results.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.05M	24	24	1536	512	3072	$\sqrt{2}$	$\sqrt{6}$	128	64	9448

Appendix

Table 10: Model configuration of MLA for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d$	$d_{h}$	$d_{f}$
2.9B	2873.23M	24	24	1	2048	3072	256	8024

Appendix

Table 11: Model configuration of MFA for main results.

Model Size	# Parameters	# Layers	$h$	$\beta_{q}$	$\beta_{kv}$	$d$	$d_{h}$	$d_{f}$
2.9B	2873.18M	24	24	6	2	3072	128	10760

Appendix

Table 12: Model configuration of TPA for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	2	1024	512	3072	$\sqrt{3}$	$\sqrt{12}$	128	64	10048

Appendix

Table 13: Model configuration of GLA-2 for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.22M	24	24	4	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	128	64	10136

Appendix

Table 14: Model configuration of GLA-4 for main results.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.00M	24	24	6	3072	128	64	9960

Appendix

Table 15: Model configuration of GTA for main results.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$\alpha_{attn}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	$\frac{\sqrt{2}}{2}$	128	64	10048

Appendix

Table 16: Model configuration of MLRA-2 for main results.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$\alpha_{attn}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.22M	24	24	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	$\frac{1}{2}$	128	64	9880

Appendix

Table 17: Model configuration of MLRA-4 for main results.

Model Size	# Parameters	# Layers	$h$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	24	3072	128	8192

Appendix

Table 18: Model configuration of MHA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.00M	24	24	1	3072	128	10152

Appendix

Table 19: Model configuration of MQA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	24	6	3072	128	9728

Appendix

Table 20: Model configuration of GQA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.05M	24	24	1536	512	3072	$\sqrt{2}$	$\sqrt{6}$	128	64	9448

Appendix

Table 21: Model configuration of MLA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d$	$d_{h}$	$d_{f}$
2.9B	2873.23M	24	24	1	2048	3072	256	8024

Appendix

Table 22: Model configuration of MFA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$\beta_{q}$	$\beta_{kv}$	$d$	$d_{h}$	$d_{f}$
2.9B	2873.18M	24	24	6	2	3072	128	10760

Appendix

Table 23: Model configuration of TPA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	2	1024	512	3072	$\sqrt{3}$	$\sqrt{12}$	128	64	10048

Appendix

Table 24: Model configuration of GLA-2 for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.22M	24	24	4	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	128	64	10136

Appendix

Table 25: Model configuration of GLA-4 for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.00M	24	24	6	3072	128	64	9960

Appendix

Table 26: Model configuration of GTA for initialization ablation.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.05M	24	24	1536	512	3072	$1$	$1$	128	64	9448

Appendix

Table 27: Model configuration of MLA in the absence of scaling factors.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	2	1024	512	3072	$1$	$1$	128	64	10048

Appendix

Table 28: Model configuration of GLA-2 in the absence of scaling factors.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$\alpha_{attn}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	1024	512	3072	$1$	$1$	$1$	128	64	10048

Appendix

Table 29: Model configuration of MLRA-2 in the absence of scaling factors.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	48	6	3072	128	7680

Appendix

Table 30: Model configuration of GQA parameterized with

2\times

attention heads.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.23M	24	48	1536	512	3072	$\sqrt{2}$	$\sqrt{6}$	128	64	7320

Appendix

Table 31: Model configuration of MLA parameterized with

2\times

attention heads.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.22M	24	48	2	1024	512	3072	$\sqrt{3}$	$\sqrt{12}$	128	64	8344

Appendix

Table 32: Model configuration of GLA-2 parameterized with

2\times

attention heads.

Model Size	# Parameters	# Layers	$h$	$g$	$d$	$d_{h}$	$d_{f}$
2.9B	2872.59M	24	24	6	3072	128	8704

Appendix

Table 33: Model configuration of GQA incorporating gated attention.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.05M	24	24	1536	512	3072	$\sqrt{2}$	$\sqrt{6}$	128	64	8424

Appendix

Table 34: Model configuration of MLA incorporating gated attention.

Model Size	# Parameters	# Layers	$h$	$g$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	2	1024	512	3072	$\sqrt{3}$	$\sqrt{12}$	128	64	9024

Appendix

Table 35: Model configuration of GLA-2 incorporating gated attention.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$\alpha_{attn}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2872.63M	24	24	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	$\frac{\sqrt{2}}{2}$	128	64	9024

Appendix

Table 36: Model configuration of MLRA-2 incorporating gated attention.

Model Size	# Parameters	# Layers	$h$	$d_{c}^{\prime}$	$d_{c}$	$d$	$\alpha_{q}$	$\alpha_{kv}$	$\alpha_{attn}$	$d_{h}$	$d_{h}^{R}$	$d_{f}$
2.9B	2873.22M	24	24	1024	512	3072	$\sqrt{3}$	$\sqrt{24}$	$\frac{1}{2}$	128	64	8856

Appendix

Table 37: Model configuration of MLRA-4 incorporating gated attention.

Method	Initialization	Wikipedia	C4	Pile	RefinedWeb	Cosmopedia	FineWeb	FineWeb-Edu	Avg
MHA	$\mathcal{N}(0,\sigma=0.02)$	14.759	16.800	13.282	18.988	9.356	15.904	9.571	14.094
MHA	zero	14.624	16.575	12.929	18.698	9.102	15.656	9.434	13.860
MQA	$\mathcal{N}(0,\sigma=0.02)$	14.708	17.075	13.500	19.301	9.510	16.190	9.697	14.283
MQA	zero	15.134	16.837	14.008	19.202	9.484	15.942	9.533	14.306
GQA	$\mathcal{N}(0,\sigma=0.02)$	14.687	16.882	13.528	19.084	9.422	15.974	9.571	14.164
GQA	zero	15.057	16.628	13.758	18.885	9.504	15.713	9.427	14.139

Appendix

Table 38: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, RefinedWeb, Cosmopedia, FineWeb, and FineWeb-Edu. We compare two initialization strategies, zero versus Gaussian (

\mathcal{N}(0,\sigma=0.02)

), applied to the output projection weights

\bm{W}^{\text{O, attn}}

and

\bm{W}^{\text{O, mlp}}

Method	Scaling	Wikipedia	C4	Pile	RefinedWeb	Cosmopedia	FineWeb	FineWeb-Edu	Avg
MLA	w/o	14.461	16.386	13.218	18.636	8.961	15.485	9.307	13.779
MLA	w/	14.567	16.345	12.965	18.523	8.966	15.440	9.284	13.727
GLA-2	w/o	14.518	16.467	13.179	18.612	9.138	15.565	9.305	13.827
GLA-2	w/	14.605	16.323	13.225	18.509	9.118	15.424	9.249	13.779
MLRA-2	w/o	14.326	16.485	13.145	18.657	9.168	15.570	9.304	13.808
MLRA-2	w/	14.615	16.342	13.236	18.602	9.153	15.439	9.242	13.804

Appendix

Table 39: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, RefinedWeb, Cosmopedia, FineWeb, and FineWeb-Edu. This analysis specifically compares models without and with scaling.

Method	$2\times$ Attention Heads	Wikipedia	C4	Pile	RefinedWeb	Cosmopedia	FineWeb	FineWeb-Edu	Avg
GQA	w/	15.280	16.702	13.789	18.961	9.486	15.785	9.490	14.213
GQA	w/o	15.057	16.628	13.758	18.885	9.504	15.713	9.427	14.139
MLA	w/	14.771	16.432	13.108	18.615	9.029	15.529	9.371	13.836
MLA	w/o	14.567	16.345	12.965	18.523	8.966	15.440	9.284	13.727
GLA-2	w/	14.969	16.313	13.428	18.569	8.991	15.410	9.281	13.851
GLA-2	w/o	14.605	16.323	13.225	18.509	9.118	15.424	9.249	13.779

Appendix

Table 40: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, RefinedWeb, Cosmopedia, FineWeb, and FineWeb-Edu. This analysis specifically compares models with and without

2\times

attention heads.

Figure 7: Training loss curves for all models.

Explore similar work

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
Mar 18, 2026Zhongzhu Zhou, Fengxiang Bie, Ziyan Chen +6Multi-Head Latent Attention Low-Rank Approximation
University of Sydney · Together AI · University of Texas at Austin +1
Hybrid Latent Attention for Looped Language Models
Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model Inference LLM Inference Acceleration
Virginia Tech · Independent Researcher

May 14, 2026J/K move · Enter open · S save

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

Fanxu Meng, Zhixin Pan

Institute for Artificial Intelligence, Peking University

Grouped-Query Attention GPU Acceleration Self-Attention Transformer Attention LLM Inference Acceleration KV-Cache Compression+2

Open paper AskPDF arXiv

Decoding efficiency depends on KV-cache traffic, computation, and parallelism. Multi-head Latent Attention (MLA) offers absorbed and head-expanded forms, but its head-specific projections limit grouped KV sharing. We introduce Group-Query Latent Attention (GQLA), whose group-shared up-projections preserve a joint latent and enable a shardable expanded cache. The same weights support equivalent MQA-absorb and GQA execution using existing kernels. Roofline analysis relates path preference to GPU compute-to-bandwidth ratio and tensor parallelism (TP): absorption trades traffic for computation and replicates the latent across ranks; GQA shards KV groups. Pretraining with eight query heads per group yields terminal training losses close to MLA's across the tested model scales at matched tokens. TransGQLA converts GQA and MLA checkpoints using calibration data. Without gradient updates, converted DeepSeek-V3.1-Base achieves 77.79% eight-task accuracy versus the source's 77.84%, while WikiText-2 perplexity rises from 3.21 to 6.05. On eight H20-3e GPUs, its GQA path improves output throughput over MQA-absorb by 22.42--34.97% at matched resident batches across 8K--64K inputs. Its smaller per-rank cache permits larger batches, giving 50.07% in a separate capacity-tuned comparison. These results support multiple execution paths as an architectural means of adapting decoding to hardware and parallelism.

Abstract

Explore similar work

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention

Hybrid Latent Attention for Looped Language Models