PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models

Junjie Cao; Fengping Tian; Wenjie Zhou; Xuanfan Ni; Chenyang Lyu; Siyou Liu; Longyue Wang; Weihua Luo

eess.ASOct 15, 2026

PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models

Authors: Junjie Cao, Fengping Tian, Wenjie Zhou, Xuanfan Ni, Chenyang Lyu, Siyou Liu, Longyue Wang, Weihua Luo

Organizations: Alibaba, Hangzhou, China · University of Macau

Abstract

Sparse routing reduces per-token computation but leaves the full expert bank resident in memory. We study expert pruning for speech-centric omni-modal models, where pooling calibration statistics can obscure the different expert demands of acoustic-input processing and text-answer generation. We propose PW-Pruning, a phase-aware extension of DO-ACP that constructs separate importance-weighted kernels for acoustic-prefill and teacher-forced answer tokens, then combines phase-weighted log-determinant marginal gains to select one static expert bank. On Qwen3-Omni, retaining 32 of 128 experts per layer improves MMAU-mini and MMSU accuracy over matched pooled selection by 3.20 and 4.38 percentage points, respectively, after supervised fine-tuning, averaged over three runs. Single-phase and shuffled-label controls on this public backbone suggest that semantic phase information contributes to these gains. Experiments on a non-public Qwen3.5-Omni backbone and English-only zero-shot evaluation of Nemotron provide complementary evidence. Across the tested Qwen budgets, AU gains are larger under stronger pruning, without inference-time mask switching.

Figures & tables

Panel of Figure 1 — Figure 1: From phase demand to one static mask. Qwen3-Omni: (a) audio-prefill/teacher-forced-answer frequency-mask Jaccard. Legends: means; shading: last eight layers; dotted: $B/(2E-B)$ cardinality guides. (b) Actual E32 mask assignment coverage, task/layer macro-averaged on calibration. (c) Phase-wise selection. E: experts per MoE layer; normalized depth.

Zero-shot
Budget	Selector	CV-W $\downarrow$	CV-C $\downarrow$	LS $\downarrow$	OCP $\downarrow$	BLEU $\uparrow$	MMAU $\uparrow$	MMAR $\uparrow$	MMSU $\uparrow$
E128	Teacher	6.17	6.42	2.11	1.49	29.34	78.00	70.30	71.04
E64	Pooled	10.6	9.0	3.1	2.9	18.4	72.2	64.6	67.8
E64	PW	11.3	9.9	3.2	2.2	18.4	73.4	63.9	67.5
E64	PS	23.3	6.9	3.0	2.7	17.7	71.6	64.3	66.0
E64	REAP	22.2	10.7	3.2	3.9	15.2	72.2	63.5	66.7

Table 1: Speech-centric pruning on Qwen3-Omni. CV-W/CV-C: Common Voice en+fr micro-WER / yue+zh micro-CER; LS: LibriSpeech WER; OCP: OpenCpop CER (%); AU: accuracy (%). E: retained experts. SFT: mean

\pm

s.d.; bold: better Pooled/PW mean.

Panel of Figure 2 — Figure 2: Phase-semantics recovery on Qwen3-Omni, E32 . All methods use the same expert budget and SFT schedule. Final Pooled/PW error bars show three-run sample s.d. (Table 1 ); mean control endpoints are listed in Table 2 .

Qwen3-Omni, E32 (Fig. 2 )
Selector	MMAU $\uparrow$	MMAR $\uparrow$	MMSU $\uparrow$
Shuffled-PW	63.21	50.20	53.20
Audio-only	64.72	50.95	53.36
Answer-only	61.96	52.95	53.59
Qwen3.5-Omni, $4\times$ reduction
Pooled	70.40	62.60	66.02

Table 2: Phase-semantics ablation. Mean AU accuracy (%) across recovery runs at SFT step 1,024. Qwen3-Omni E32 Pooled/PW: Table 1 ; Qwen3.5-Omni

4\times

Pooled/PW: same means as Table 3 . Bold: highest mean per column in the Qwen3.5-Omni block.

$R$	Selector	CV-W $\downarrow$	CV-C $\downarrow$	LS $\downarrow$	OCP $\downarrow$	BLEU $\uparrow$	MMAU $\uparrow$	MMAR $\uparrow$	MMSU $\uparrow$
$1\times$	Teacher	5.68	5.70	2.26	1.06	30.72	78.50	73.60	71.10
$2\!\times$	Pooled	9.67 $\pm$ .04	8.77 $\pm$ .28	3.01 $\pm$ .00	2.10 $\pm$ .03	26.14 $\pm$ .25	75.70 $\pm$ .36	70.40 $\pm$ .26	69.83 $\pm$ .21
$2\!\times$	PW	9.96 $\pm$ .15	8.76 $\pm$ .48	3.05 $\pm$ .03	2.64 $\pm$ .57	25.89 $\pm$ .03	76.20 $\pm$ .62	70.17 $\pm$ .71	69.49 $\pm$ .44
$4\!\times$	Pooled	16.48 $\pm$ .22	14.75 $\pm$ .20	5.04 $\pm$ .11	5.22 $\pm$ .77	12.81 $\pm$ .23	70.40 $\pm$ .14	62.60 $\pm$ .57	66.02 $\pm$ .42
$4\!\times$	PW	14.34 $\pm$ .48	14.73 $\pm$ 1.08	4.66 $\pm$ .08	4.52 $\pm$ .41	13.65 $\pm$ .24	72.50 $\pm$ .71	63.05 $\pm$ 1.06	66.45 $\pm$ .47

Table 3: Transfer to Qwen3.5-Omni. A non-public research backbone; budgets are relative to the unpruned model. Matched SFT recovery: mean

\pm

sample s.d. across repeated runs. Metrics follow Table 1 . Bold: better mean within each Pooled/PW pair.

Budget	Selector	En-WER $\downarrow$	MMAU $\uparrow$	MMAR $\uparrow$	MMSU $\uparrow$
E128	Teacher	3.378	77.60	60.70	62.60
E64	Pooled	4.173	75.00	58.50	60.00
E64	PW	4.002	74.30	59.10	61.08
E32	Pooled	5.535	69.50	52.10	50.58
E32	PW	4.797	69.50	52.50	52.54

Table 4: Zero-shot Nemotron transfer. English-subset public ASR WER and AU accuracy (%); E: retained experts. No recovery; bold: better Pooled/PW score.

Panel of Figure 3 — Figure 3: Deployment savings. H20, batch 1: (a) peak memory relative to the unpruned model; (b) real-time factor. Savings reflect expert budget, not PW-specific speedups.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 58 papers

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts Pruning Sparse Mixture-of-Experts
Intellifusion Inc.
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Jul 2, 2026Yongqin Zeng, Sicheng Pan, Jiale Wang +4Mixture-of-Experts Pruning LLM Pruning

Oct 19, 2026J/K move · Enter open · S save

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao+6

LinkedIn · MIT · Google

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression Neural Network Pruning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.

Abstract

Mixture-of-Experts Pruning

Explore similar work

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models