Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.
Figures & tables
Figure 1: DuplexTune architecture. (a) Adapted decoder stack. (b) Width-lift and width-return bridges connect paired states to pretrained attention and sparse experts. Attention updates a to u ; expert processing then updates b to v before the channel join. (c) Schematic expert bank with frozen router weights during backbone adaptation.
Method
Memory (GB) ↓
Samples/s ↑
Parameter-efficient adaptation
LoRA
18.4
76.2
DoRA
19.8
72.5
(IA)3
17.6
74.8
Broader weight updates
SFT + checkpointing
64.8
19.9
Table 1: Single-H800 efficiency. Batch sizes are capacity-tuned; throughput is not a matched-token comparison.
Method
MMLU
GSM8K
Multi.
MT-B.
Base model
62.4
61.2
40.4
6.25
LoRA
65.5
71.8
38.7
7.21
DoRA
65.9
71.1
39.1
7.29
(IA)3
65.2
70.5
38.4
7.18
SFT + checkpointing
66.3
75.1
39.8
7.56
LOMO
66.5
74.9
39.5
7.52
Table 2: Downstream scores. MMLU, GSM8K, and Multi. are percentages; MT-B. is MT-Bench. Bold marks the best score per column, including the base model.
Figure 2: Memory and MMLU. Points combine the values in Tables 1 and 2 ; they summarize complete configurations with capacity-tuned batches. Throughput and other task scores remain separate criteria.
Configuration
MMLU (%) ↑
DuplexTune (complete)
67.2
No bridge warm-up (joint updates)
57.5
No backbone adaptation (bridges only)
54.9
Table 3: Training-phase ablation on MMLU. Removing either alignment or backbone fine-tuning reduces the score.
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.
Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling. We propose Mixture of Reused Experts (MoRE), a hybrid that shares expert pools across groups of adjacent layers. Each layer retains its own router but selects from a larger shared pool, expanding the diversity of routing combinations without additional parameters. To enable shared experts to distinguish between layers, we introduce lightweight learnable depth embeddings that condition each layer's input before routing. Experiments across three model scales (114M-1.15B parameters) show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.
Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4
Department of Computer Science, Cornell University
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7