Dynamic Expert Pruning for Multi-Agent Systems

Jabin Koo; Soheil Abbasloo; Sungjae Lee; Jungseul Ok

cs.LGOct 2, 2026

Dynamic Expert Pruning for Multi-Agent Systems

Authors: Jabin Koo, Soheil Abbasloo, Sungjae Lee, Jungseul Ok

Organizations: Pohang University of Science and Technology (POSTECH) · Microsoft Research Asia

Abstract

Mixture-of-Experts (MoE) architectures scale language models efficiently by activating only a few experts per token, but the saving is confined to computation: every expert must stay resident on the accelerator, so memory bounds where these models can be deployed. Expert pruning reduces this footprint, yet existing methods are static --- a single mask, calibrated offline, is applied to the model for every subsequent request. This assumption can fail when the workload is heterogeneous, most prominently in multi-agent systems, where one backbone serves many tasks and roles at once: our analysis shows that different tasks and roles recruit different experts, while static methods assign one fixed subset to all of them. We therefore propose Dynamic Expert Pruning (DEP), which rests on a finding we establish here: an agent's system and task prompts are by themselves sufficient to identify the experts that agent and its task require, since that text already describes what the agent will do. A lightweight predictor, trained once on workflow transcripts, turns those prompts into a specialized per-request mask in a single forward pass, with no per-configuration calibration. Across diverse tasks and roles, model scales, and MoE architectures, DEP achieves better overall accuracy than static pruning and merging baselines, and generalizes to workflows unseen in training without retraining. Its margin over those baselines is largest when few experts are retained, suggesting that the role specialization inherent to multi-agent systems permits sparser serving than static pruning allows.

Figures & tables

Panel of Figure 1 — Figure 1: Static masks that track the dense model for a single agent lose much of that accuracy on a multi-agent workload. Left: identical pruning methods, backbone, retention ( $\rho{=}50\%$ ), and calibration data; only the serving setting changes. Right: the resulting accuracy–memory frontier.

Figure 2: Agents differ in what they do, and therefore in the experts they use. (a) Per-role and (b) per-operator expert usage (HumanEval+, layer 24; deviation from the row mean over the top-64 experts, percentage points); (c) share of each role’s segments per operator, across three tasks.

50% retention (keep 64/128; 29.9 GB resident)
Method	MATH-500	GSM8K	HumanEval+	MBPP+	MMLU	HotpotQA	Mean
Dense (56.9 GB)	68.3 $\pm$ 8.6	97.3 $\pm$ 1.2	92.8 $\pm$ 1.4	80.0 $\pm$ 6.9	82.7 $\pm$ 1.5	71.9 $\pm$ 1.7	82.2 $\pm$ 1.0
Freq.	62.3 $\pm$ 9.0	95.3 $\pm$ 1.5	2.9 $\pm$ 1.4	3.0 $\pm$ 1.0	51.0 $\pm$ 2.6	65.4 $\pm$ 3.7	46.7 $\pm$ 1.9
REAP	40.7 $\pm$ 6.8	71.3 $\pm$ 4.5	78.7 $\pm$ 0.8	45.7 $\pm$ 4.0	40.3 $\pm$ 4.0	47.3 $\pm$ 3.0	54.0 $\pm$ 1.6
HC-SMoE	60.3 $\pm$ 6.4	84.0 $\pm$ 3.0	75.8 $\pm$ 2.2	75.0 $\pm$ 7.8	48.3 $\pm$ 0.6	60.4 $\pm$ 1.6	67.3 $\pm$ 1.5
DEP	67.3 $\pm$ 10.6	95.3 $\pm$ 2.3	84.5 $\pm$ 0.8	76.7 $\pm$ 2.5	70.7 $\pm$ 4.7	60.7 $\pm$ 3.2	75.9 $\pm$ 2.3

Table 1: DEP gives the best overall accuracy at both retentions, and is the only policy still usable at

\rho{=}25\%

. AFlow-discovered multi-agent workflows (Qwen3-30B-A3B); DEP against static masks calibrated on the identical union. Values are % (pass@1; HotpotQA SQuAD-F1; MMLU accuracy), mean

\pm

std; protocol in Section 4.1 .

			$\rho=50\%$				$\rho=25\%$
	task	Dense	Freq.	REAP	HC-SMoE	DEP	Freq.	REAP	HC-SMoE	DEP
Unseen structure	HotpotQA	68.0 $\pm$ 6.6	12.9 $\pm$ 1.5	12.3 $\pm$ 2.7	54.5 $\pm$ 5.5	65.0 $\pm$ 2.8	0.4 $\pm$ 0.4	5.3 $\pm$ 3.3	0.1 $\pm$ 0.1	38.3 $\pm$ 0.6
	HumanEval+	94.3 $\pm$ 3.5	39.0 $\pm$ 2.6	59.3 $\pm$ 1.5	82.3 $\pm$ 1.5	83.4 $\pm$ 4.1	0.0 $\pm$ 0.0	7.7 $\pm$ 2.5	0.0 $\pm$ 0.0	9.7 $\pm$ 3.2
	MATH	68.3 $\pm$ 3.3	0.0 $\pm$ 0.0	2.8 $\pm$ 2.5	45.8 $\pm$ 8.2	55.0 $\pm$ 2.4	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	9.4 $\pm$ 1.0
Unseen family	HotpotQA	67.5 $\pm$ 4.3	9.9 $\pm$ 0.1	29.5 $\pm$ 8.9	56.4 $\pm$ 7.2	53.1 $\pm$ 3.2	0.0 $\pm$ 0.0	3.1 $\pm$ 1.2	0.0 $\pm$ 0.0	44.4 $\pm$ 2.6
	HumanEval+	92.0 $\pm$ 1.9	58.7 $\pm$ 1.7	74.0 $\pm$ 2.1	87.6 $\pm$ 3.4	88.2 $\pm$ 3.9	0.0 $\pm$ 0.0	27.5 $\pm$ 3.2	0.0 $\pm$ 0.0	28.0 $\pm$ 2.3

Table 2: One predictor, fixed after training, keeps working as the workload moves away from it. Generalization ladder: one universal predictor and one static calibration, both fixed after training on a three-task workflow pool, evaluated at increasing distance from training. Accuracy % (HotpotQA F1), 30 problems per cell, mean

\pm

std over 3 seeds. Bold = best, underline = second-best pruned method per row and retention.

Panel of Figure 3 — Figure 3: Predicted masks organize themselves by task and by role. Jaccard overlap of predicted kept-expert masks at $\rho{=}50\%$ (chance $=$ 0.33). (a) Between benchmarks; (b, c) between the agents of the MATH and HumanEval+ teams. Diagonals: the same row on different problems. Measured on hand-crafted agent teams, not the discovered workflows of Table 1 .

policy	memory (GB)	accuracy	tokens/prob	generation (s)	$\Delta$ -load (s)	total (s)
dense	56.9	0.78	1905	3239	—	3239
DEP	29.9	0.76	1872	3275	+42	3317
REAP	29.9	0.53	2077	3803	0	3803
freq	29.9	0.27	3365	5457	0	5457

Table 3: A wrong mask wastes more time rambling than a right mask spends swapping experts. Efficiency at

\rho{=}50\%

over three workflows (HotpotQA, HumanEval+, MATH; 3 random 50-problem seeds), not comparable with Table 1 ; per task in Table 12 . Tokens and times are measured;

\Delta

-load is an analytic upper bound on the per-agent-step swap cost, which static masks never pay.

Table 7

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Objective	MATH	HumanEval+	HotpotQA (F1)	Mean
Dense reference	82.0	92.0	72.2	82.1
KD $+$ BCE(REAP) (ours)	76.0	90.0	71.7	79.2
LM $+$ BCE(REAP)	76.0	84.0	68.4	76.1
LM $+$ BCE(norm)	82.0	80.0	0.0	54.0
KD only ( $\alpha{=}1$ )	30.0	—	—	—

Appendix

Table 6: Objective ablation (

\rho{=}50\%

n{=}50

). Both terms of Eq. 6 are load-bearing.

Knob	Values	Outcome
$\alpha$	0, .25, .5, .75	flat ( $\pm 3$ ); $\alpha{=}1$ collapses (above)
learning rate	$3{\cdot}10^{-5}$ , $10^{-4}$ , $3{\cdot}10^{-4}$	U-shaped $\pm 5$ $\Rightarrow$ noise
final temperature $T$	.05, .15, .4	flat

Appendix

Table 7: Hyperparameter sweep (screened on MATH, the only task with headroom; paired on identical problems). All three knobs are null; apparent winners did not survive paired replication on an independent subset.

Task	Workflow	Operator sequence
HotpotQA	debate_1 r2	Custom $\to$ Custom $\to$ Custom $\to$ Custom
	debate_1 r3	Custom $\to$ Custom $\to$ Custom $\to$ Custom $\to$ Custom
	debate_1 r5	AnsGen $\to$ AnsGen $\to$ AnsGen $\to$ ScEns $\to$ Custom
	debate_1 r6	AnsGen $\to$ AnsGen $\to$ Custom $\to$ Custom
	decompose_1 r3	Custom $\to$ AnsGen $\to$ Custom $\to$ ScEns
	decompose_1 r6	Custom $\to$ AnsGen $\to$ ScEns $\to$ Custom

Appendix

Table 8: Trained-on structures (32 workflows): only these train the universal predictor.

Task	Workflow	Score	Operator sequence
HotpotQA	debate_1 r4	0.719	Custom $\to$ Custom $\to$ Custom $\to$ Custom $\to$ Custom
	decompose_2 r4	0.833	Custom $\to$ AnsGen $\to$ AnsGen $\to$ ScEns $\to$ Custom
	direct_1 r4	0.453	Custom $\to$ AnsGen
HumanEval	review_1 r2	0.900	Custom $\to$ Custom $\to$ ScEns $\to$ Test
HumanEval	selfcons_1 r6	0.960	Custom $\to$ Custom $\to$ ScEns
MATH	direct_2 r2	0.833	Custom $\to$ ScEns

Appendix

Table 9: Level 0 held-out — unseen structures (held-out rounds of families the predictor did train on).

Task	Workflow	Score	Operator sequence
HotpotQA	genverify_1 r1	0.713	AnsGen $\to$ Custom $\to$ Custom
	genverify_2 r3	0.793	AnsGen $\to$ ScEns $\to$ Custom $\to$ Custom
	genverify_2 r4	0.790	AnsGen $\to$ Custom $\to$ Custom $\to$ Custom
	genverify_2 r5	0.713	AnsGen $\to$ AnsGen $\to$ AnsGen $\to$ ScEns $\to$ Custom $\to$ Custom
HumanEval	gentestfix_1 r1	0.920	Custom $\to$ Test $\to$ Custom $\to$ Test
	gentestfix_1 r2	0.940	Custom $\to$ Custom $\to$ Test $\to$ Custom $\to$ Test

Appendix

Table 10: Level 1 held-out — unseen families (whole families never seen in training: genverify / gentestfix / solveverify ).

Level / Task	Workflow	Score	Operator sequence
L2 $\cdot$ MBPP (code)	MBPP_1 r7	0.567	CodeGen $\to$ Custom $\to$ Test $\to$ Custom
	MBPP_1 r9	0.700	CodeGen $\to$ Test $\to$ Custom
	MBPP_1 r14	0.700	CodeGen $\to$ ScEns $\to$ Test $\to$ Custom
L3 $\cdot$ MMLU (knowledge)	MMLU_1 r1	0.867	Custom
	MMLU_1 r2	0.567	AnsGen $\to$ Custom $\to$ Custom $\to$ ScEns
	MMLU_1 r3	0.867	AnsGen $\to$ Custom $\to$ Custom $\to$ Custom $\to$ ScEns

Appendix

Table 11: Levels 2–3 held-out — unseen tasks : MBPP = unseen code benchmark, MMLU = unseen knowledge domain.

task	policy	accuracy	tokens/prob	generation (s)
HotpotQA	dense	.71 $\pm$ .03	172 $\pm$ 8	412 $\pm$ 70
	DEP	.75 $\pm$ .09	181 $\pm$ 8	405 $\pm$ 42
	REAP	.36 $\pm$ .06	224 $\pm$ 24	780 $\pm$ 96
	freq	.46 $\pm$ .08	477 $\pm$ 28	894 $\pm$ 26
HumanEval+	dense	.94 $\pm$ .03	1156 $\pm$ 72	2548 $\pm$ 266
	DEP	.90 $\pm$ .03	1120 $\pm$ 42	2391 $\pm$ 130

Appendix

Table 12: Per-task efficiency at

\rho{=}50\%

: mean

\pm

std over 3 random 50-problem seeds, sequential contention-free runs.

50% retention (keep 80/160)
Method	MATH	GSM8K	HumanEval+	MBPP+	HotpotQA	Mean
Dense (workflow)	52.0	84.0	72.0	86.0	59.7	70.7
Direct (single call)	50.0	90.0	76.0	84.0	56.9	71.4
Freq.	33.0	71.0	17.0	5.0	39.2	33.0
REAP	32.0	68.0	38.0	56.0	38.5	46.5
DEP	34.0	74.0	46.0	66.0	38.7	51.7

Appendix

Table 13: DeepSeek-V2.5-236B (59 MoE layers, 160 routed experts, top-6, plus 2 shared experts that are always active and never pruned). Values are % (

n{=}50

problems per task; pass@1, HotpotQA token-F1, MATH/GSM8K exact match); Dense and Direct are unpruned references. Bold = best pruned method per task.

Policy	Accuracy (F1)	Tokens/prob	ms/token	Median tok	Max tok	# exploded
dense	0.73	180	47.3	4	21	0
DEP	0.70	188	45.5	4	11	0
REAP	0.36	259	69.9	7	128	3
freq	0.33	510	38.9	6	128	4

Appendix

Table 14: Token statistics of the final-answer agent (HotpotQA,

\rho{=}50\%

N{=}50

). Exploded = generations

\geq

100 tokens; ms/token reflects batch padding, not per-token compute.

Policy	Output	Tok	Score
Q: “The saint in ‘A Time for Miracles’ was born on what date?”
dense	August 28, 1774	10	1.00
DEP	August 28, 1774	10	1.00
REAP	1774.1755-5.1815.1755-5.1815… (repeats to max)	128	0.00
freq	Answer: (empty)	3	0.00
Q: “Gail Matthius co-anchored Weekend Update with the actor who

Appendix

Table 15: Generation collapse under static masks: the same input, all four policies. Top: REAP collapses; bottom: freq collapses.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 52 papers

It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Higher-order pruning of experts in mixture-of-experts language models
Sep 16, 2026Alex M. Tseng, Prannay Kaul, Luca Zancato +2Mixture-of-Experts Pruning Mixture-of-Experts Language Models
AI Fundamental Research AWS Agentic AI

Jul 9, 2026J/K move · Enter open · S save

It Takes a MAESTRO To Prune Bad Experts

Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · NVIDIA, India

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression Activation Sparsity Structured Sparsity+1

Open paper AskPDF arXiv

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Abstract

Mixture-of-Experts Pruning

Explore similar work

It Takes a MAESTRO To Prune Bad Experts

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Higher-order pruning of experts in mixture-of-experts language models