MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Minsang Kim; Seungjun Baek

cs.LGOct 21, 2026

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Authors: Minsang Kim, Seungjun Baek

Organizations: Korea University · SK Telecom

Abstract

Mixture-of-Experts (MoE) language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length. Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality. We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions. MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures. Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3 $\times$ and 4.9 $\times$ higher scores, respectively.

Figures & tables

Figure 1: Router instability is associated with answer-quality degradation. As KV compression becomes more aggressive, expert selection changes more often, and performance drops accordingly.

Figure 2: Overview of MoEZip. MoEZip estimates routing fragility from instruction-token router states, transfers it to context positions via instruction-to-context attention, and reserves the top-scoring positions as routing-aware anchors. The base scorer fills the remaining budget, and thus the final cache size and decoding procedure remain unchanged.

Figure 3: Performance on 12 LongBench tasks across different KV cache ratios. MoEZip consistently preserves higher task performance than existing KV compression baselines, especially under aggressive compression.

Figure 4: Performance heatmaps over context length and document depth across different KV cache ratios. We compare MoEZip, KVZip, SnapKV, and PyramidKV under KV cache keep ratios from 0.1 to 0.9.

Figure 5: Average performance on 13 RULER tasks across different KV cache ratios.

Figure 6: Generalization to sigmoid-routed MoE models. On GLM-4.7-Flash, MoEZip consistently achieves the best average performance across 12 LongBench tasks, with the largest gains under aggressive KV compression.

Method	Reusable cache	Keep 0.2	Keep 0.1
KVZip	Yes	29.25	19.51
MoEZip-Recon	Yes	45.14	41.95
MoEZip	No	49.59	44.89

Table 1: Query conditioning and cache reuse. LongBench 12-task macro scores on Qwen3-30B-A3B-Instruct-2507 at matched KV budgets. KVZip and MoEZip-Recon share a fixed reconstruction query.

Panel of Figure 7 — Figure 7: Average performance over 12 tasks across different KV cache ratios. a)–(d) present ablations on the main components of MoEZip, including Fisher-based fragility modeling, adaptive budget allocation, head-wise aggregation, and the choice of base scorer. The full method consistently performs best, especially in the high-compression regime.

Metric	Retained set	Full KV	Compressed	Oracle routing	$\bm{\Delta}$
NLL $\downarrow$	KVZip	4.22	5.67	4.97	$-0.70$
	MoEZip	4.22	4.79	4.52	$-0.27$
LongBench $\uparrow$	KVZip	53.7	19.5	29.7	$+10.2$
	MoEZip	53.7	44.9	46.2	$+1.3$

Table 9

Corrected sites	Answer NLL $\downarrow$	Normalized recovery
0%	5.670	0.0%
25%	$5.503\pm 0.015$	23.8%
50%	$5.273\pm 0.012$	56.7%
75%	$5.107\pm 0.008$	80.5%
100%	4.970	100.0%

Table 2: Progressive routing correction. The retained KVZip cache is fixed; recovery is normalized between zero and complete correction.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Empirical expert-flip rate across Fisher-trace quantiles. We bin

(\text{sample},\text{decode step},\text{layer})

observations by Fisher trace from the full-KV router softmax and measure the mean top-1 expert-flip rate under compressed KV. Across both HotpotQA and Qasper, higher Fisher-trace bins consistently show higher flip rates, and the separation becomes larger as compression becomes more aggressive.

Task	Q1	Q2	Q3	Q4	Q5	Q5/Q1
MuSiQue	.103	.144	.223	.275	.429	4.17
2WikiMQA	.085	.127	.168	.210	.340	4.00
PsgRetrieval	.142	.207	.265	.369	.503	3.54
RepoBench-P	.185	.302	.375	.475	.609	3.29
MultiFieldQA	.217	.337	.455	.556	.704	3.24
Qasper	.230	.355	.472	.575	.723	3.14

Appendix

Table 3: Realized expert-flip rates by Fisher-trace quintile. Qwen3-30B with KVZip at keep ratio

0.1

. Q1 and Q5 denote the lowest and highest quintiles. Ratios are computed from the displayed rounded flip rates.

Dataset	Keep ratio	Spearman $\rho$	Pearson $r$
HotpotQA	0.7	0.26	0.28
HotpotQA	0.5	0.31	0.39
HotpotQA	0.3	0.50	0.54
HotpotQA	0.1	0.70	0.68
Qasper	0.7	0.41	0.40
Qasper	0.5	0.54	0.52

Appendix

Table 4: Sample-level correlation between answer-token Fisher trace and expert flip rate. For each sample, we compute the mean Fisher trace from full-KV answer-token router states and correlate them with the expert flip rate under compressed KV. Both Spearman’s

\rho

and Pearson’s

r

are positive across all settings and increase under more aggressive compression, indicating that the Fisher-trace proxy captures sample-level routing fragility.

Dataset	$\rho(X_{1},X_{2})$	$\rho(X_{1},Y)$		$\rho(X_{2},Y)$
Dataset	$\rho(X_{1},X_{2})$	Keep 0.1	Keep 0.3	Keep 0.1	Keep 0.3
QMSUM	$+0.351^{*}$	$+0.103$	$+0.367^{*}$	$+0.105$	$+0.215$
MultiFieldQA	$+0.475^{*}$	$+0.351^{*}$	$+0.242$	$+0.825^{*}$	$+0.563^{*}$

Appendix

Table 5: Instruction-token surrogate validation.

X_{1}

: pre-generation instruction-token mean Fisher score (observable at eviction time).

X_{2}

: answer-token mean Fisher score under full KV (the quantity

X_{1}

is meant to surrogate).

Y

: compressed-KV expert flip rate (a downstream, highly aggregated statistic). Load-bearing claim (

X_{1}\leftrightarrow X_{2}

): the surrogate relation is significant across all tasks (

p<0.05

) and is independent of the keep ratio (both are computed under full KV). Supporting evidence (

X_{1},X_{2}\leftrightarrow Y

): Fisher scores additionally correlate with the downstream flip rate, with strength varying by task structure. Asterisks denote correlations significant at

p<0.05

Task	KVZip	Instruction	Answer oracle	Recovery	Jaccard
PsgRetrieval	3.5	63.3	65.1	.97	.96
HotpotQA	22.8	48.6	52.3	.87	.88
Qasper	8.6	18.3	31.2	.43	.71
QMSUM	16.8	17.5	20.3	.20	.64
Macro / aggregate	12.9	36.9	42.2	.82	.80

Appendix

Table 6: Instruction-conditioned versus answer-oracle selection. Qwen3-30B at keep ratio

0.1

. The final row reports macro task scores, aggregate recovery from Eq. ( 35 ), and mean Jaccard similarity.

Task	Keep	KVZip	Instruction	Oracle	Recovery	Jaccard
NIAH-single-2	.2	15.4	100.0	100.0	1.00	.94
NIAH-multikey-1	.2	10.6	86.0	94.0	.90	.76
NIAH-multiquery	.2	10.0	77.0	77.0	1.00	.81
NIAH-single-2	.1	10.2	72.0	88.0	.79	.52
NIAH-multikey-1	.1	10.4	44.0	60.0	.68	.44

Appendix

Table 7: Answer-oracle diagnostic on RULER/NIAH. All settings use 4K-token contexts. Instruction and oracle conditions use matched retained-KV and anchor budgets.

Figure 9: Softmax-router worst-case bound versus Fisher trace. The worst-case Fisher-bound score performs comparably to Fisher trace on several tasks, but does not consistently improve performance and can be less robust under aggressive compression. We therefore use Fisher trace as the default routing-fragility proxy.

Figure 10: Sigmoid-router worst-case score versus Fisher trace. For sigmoid-style routing, the Fisher matrix is diagonal and the worst-case curvature is given exactly by $\max_{e}g_{e}(1-g_{e})$ . We compare this exact worst-case score with the Fisher trace on six LongBench tasks. The two scores show similar trends overall, but the worst-case score does not consistently outperform Fisher trace and can degrade more sharply in the high-compression regime.

Method	Avg. Scoring Time (s)	Additional Time (s)
KVZip (base)	9.36	-
MoEZip (+Fisher score)	10.24	0.88

Appendix

Table 8: Scoring-time overhead of MoEZip on Qasper. We compare the average scoring time of the base KVZip scorer and MoEZip under the same evaluation setting. The additional overhead of Fisher-based scoring remains modest.

Figure 11: Scoring-time overhead on NIAH. Adding the proposed Fisher-based routing refinement to KVZip introduces only modest additional scoring cost on Qwen3-30B-A3B at a KV cache ratio of 0.5, with absolute overhead ranging from 0.27s to 1.79s across context lengths.

Figure 12: Sensitivity analysis of $\rho_{\max}$ on representative LongBench tasks. We vary the maximum anchor ratio $\rho_{\max}\in\{0.1,0.25,0.3\}$ while fixing $\rho_{\min}=0.05$ . Across QMSum, Qasper, MultiFieldQA, and 2WikiMQA, the overall performance is relatively stable over a broad range of $\rho_{\max}$ values. The default setting $\rho_{\max}=0.25$ consistently provides strong or near-best performance, while overly small values can slightly weaken robustness in the high-compression regime by limiting the number of route-aware anchors.

Figure 13: Sensitivity analysis of $\rho_{\min}$ on representative LongBench tasks. We vary the minimum anchor ratio $\rho_{\min}\in\{0.0,0.05,0.1\}$ while fixing $\rho_{\max}=0.25$ . The results remain highly stable across all four tasks, indicating that MoEZip is not overly sensitive to the exact choice of $\rho_{\min}$ . The default setting $\rho_{\min}=0.05$ achieves consistently strong performance, while different values lead only to minor variations across KV cache ratios.

Figure 14: Task quality and router flip rate across compression ratios. We compare task performance and expert flip rate of answer-tokens under increasing KV compression. Across all six tasks, MoEZip consistently maintains lower flip rates than KVZip. The shaded region marks the compression range where KVZip begins to exhibit a sharp quality drop, while MoEZip remains notably more stable.

Comparison	Keep 0.1	Keep 0.3	Keep 0.5
MoEZip $-$ KVZip	$+25.4$ [ $+13.5$ , $+36.9$ ]	$+8.7$ [ $+5.3$ , $+11.9$ ]	$+0.58$ [ $+0.17$ , $+1.12$ ]
MoEZip $-$ SnapKV	$+13.8$ [ $+7.8$ , $+21.1$ ]	$+6.1$ [ $+3.3$ , $+9.4$ ]	$+3.3$ [ $+1.8$ , $+4.9$ ]
MoEZip $-$ PyramidKV	$+16.3$ [ $+9.2$ , $+25.4$ ]	$+14.5$ [ $+6.7$ , $+25.2$ ]	$+13.9$ [ $+7.5$ , $+21.2$ ]

Appendix

Table 9: Paired task-level bootstrap intervals for LongBench. Each entry reports the mean score difference between MoEZip and a baseline, with 95% percentile intervals over 10,000 bootstrap samples. All intervals exclude zero, indicating consistent improvements under task-level resampling.

Comparison	Keep 0.1	Keep 0.3	Keep 0.5
MoEZip $-$ KVZip	$+20.4$ [ $+9.2$ , $+34.9$ ]	$+16.1$ [ $+6.3$ , $+27.9$ ]	$+5.6$ [ $+0.2$ , $+13.3$ ]
MoEZip $-$ SnapKV	$+9.7$ [ $+1.6$ , $+25.1$ ]	$+55.1$ [ $+35.3$ , $+73.9$ ]	$+53.5$ [ $+33.7$ , $+72.1$ ]
MoEZip $-$ PyramidKV	$+9.8$ [ $+1.6$ , $+25.1$ ]	$+60.4$ [ $+42.5$ , $+76.8$ ]	$+71.5$ [ $+54.3$ , $+87.6$ ]

Appendix

Table 10: Paired task-level bootstrap intervals for RULER. Each entry reports the mean score difference between MoEZip and a baseline, with 95% percentile intervals over 10,000 bootstrap samples. All intervals exclude zero, indicating consistent improvements under task-level resampling.

Method	Reusable cache	Keep 0.2	Keep 0.1
KVZip	Yes	29.25	19.51
MoEZip-Recon	Yes	45.14	41.95
MoEZip	No	49.59	44.89

Appendix

Table 11: Query conditioning and compressed-cache reuse. LongBench 12-task macro scores on Qwen3-30B-A3B-Instruct-2507. All methods use matched retained-KV budgets.

Evaluation	Global	Per-layer	Per-GQA
LongBench 12-task macro	44.9	36.3	39.3
Gap relative to global	—	$-8.6$	$-5.6$
Tasks won by global	—	12/12	11/12

Appendix

Table 12: Anchor-selection granularity under matched KV budgets. The base compressor retains layer/head-specific selection for all non-anchor entries.

Task	Overlap	New / 128
RULER QA-1	15%	108
NIAH-single-2	16%	108
NIAH-multikey-1	16%	107
NIAH-multiquery	16%	107
LCC	17%	106
TREC	20%	103

Appendix

Table 13: Layer–head-aware anchor overlap at

B_{f}=128

. New / 128 is the mean number of anchor positions absent from the corresponding KVZip cache. Percentages and counts are rounded independently.

Method	HotpotQA	PsgRet	Qasper	QMSUM	Macro
Uncompressed	57.0	100.0	46.9	23.6	56.9
KVZip	35.2	4.0	10.9	16.8	16.7
Attention-only	31.4	38.8	11.0	15.0	24.1
MoEZip	45.6	61.5	17.5	17.3	35.5

Appendix

Table 14: Four-task evaluation on Qwen3-235B-A22B. Compressed methods use keep ratio

0.1

. Bold indicates the best compressed result; the uncompressed model is a reference.

Task	KVZip	MoEZip	$\bm{\Delta}$
PsgRet	4.0	61.5	$+57.5$
TREC	16.0	50.0	$+34.0$
LCC	28.8	55.9	$+27.1$
MultiFieldQA	20.7	35.6	$+14.8$
TriviaQA	75.5	86.9	$+11.4$
HotpotQA	35.2	45.6	$+10.5$

Appendix

Table 15: Ten-task evaluation on Qwen3-235B-A22B at keep ratio

0.1

. Deltas use unrounded task scores and can differ slightly from subtraction of displayed values. This task set differs from the four-task control comparison in Table 14 .

Diagnostic	Qwen3-30B	Qwen3-235B
Fisher Q5/Q1 flip-rate ratio	$2.8\times$	$2.5\times$
Anchor Jaccard: HotpotQA	.88	.92
Anchor Jaccard: PsgRetrieval	.96	.97

Appendix

Table 16: Routing-signal diagnostics across model scales. Results at keep ratio

0.1

. Fisher quintiles are formed within each task and layer, with flip rates aggregated over HotpotQA and PassageRetrieval-en. Jaccard similarity compares instruction- and answer-conditioned anchor sets.

Method / quantity	BF16	INT8	Change
KVZip	12.8	12.5	$-0.3$
MoEZip	26.4	25.8	$-0.6$
MoEZip $-$ KVZip	$+13.6$	$+13.3$	$-0.3$

Appendix

Table 17: Composition with INT8 KV quantization. Six-task LongBench macro scores at keep ratio

0.1

. Within each method, BF16 and INT8 retain the same context positions; anchors receive the same quantization treatment as other retained entries.

Conditional probability	Value
$\Pr(F_{t,\ell+1}=1\mid F_{t,\ell}=1)$	.448
$\Pr(F_{t,\ell+1}=1\mid F_{t,\ell}=0)$	.270

Appendix

Table 18: Association between routing flips at adjacent MoE layers. Results at KV keep ratio

0.1

. Conditional probabilities are computed from pooled answer-token observations across all evaluated tasks and adjacent MoE-layer pairs.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM Inference KV Caching
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Nov 6, 2025Yushu Zhao, Zheng Wang, Minjia ZhangLLM Compression Mixture-of-Experts Models
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.

Apr 20, 2026J/K move · Enter open · S save

MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression

Libo Sun, Peixiong He, Po-Wei Harn, Xiao Qin

Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan

LLM Inference KV Caching Memory-Efficient Optimization KV-Cache Eviction KV-Cache Compression KV-Cache Quantization

Open paper AskPDF arXiv

KV cache memory is the dominant bottleneck for long-context LLM inference. Existing compression methods each act on a single axis of the four-dimensional KV tensor -- token eviction (sequence), quantization (precision), low-rank projection (head dimension), or cross-layer sharing -- but apply the same recipe to every layer. We show that this homogeneity leaves accuracy on the table: different layers respond very differently to each compression operation, and the optimal per-layer mix of eviction and quantization is far from uniform. We propose MoE-nD, a mixture-of-experts framework that routes each layer to its own (eviction-ratio, K-bits, V-bits) tuple under a global memory budget. An offline-calibrated greedy solver chooses the routing that minimizes predicted quality loss; at inference time, per-layer heterogeneous eviction and quantization are applied jointly through a single attention patch. On a 4-task subset of LongBench-v1 (16k inputs, n=50 per task, adapted reasoning-model protocol; see section Experiments), MoE-nD's hetero variant matches our uncompressed 1.9~~GB baseline at 14x compression (136~~MB) while every other compressed baseline we tested (1d, 2d_uniform, 2d) at comparable or smaller memory stays under 8/100. The gains hold on AIME reasoning benchmarks (+6 to +27 pts over the strongest per-layer-quantization baseline across eight configurations). Two null results -- MATH-500 and LongBench's TREC -- share a principled cause (short inputs, solver picks keep=1.0 on most layers), cleanly characterizing when per-layer eviction routing has headroom to help.

Abstract

KV-Cache Compression

Explore similar work

MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference