DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

Paul Raj; Simon Caldwell; Daniel Whitmore; Mutlu Mete

cs.LGOct 26, 2026

DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

Authors: Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete

Organizations: Texas A&M University-Commerce · Michigan State University

Abstract

Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.

Figures & tables

Figure 1: DuplexTune architecture. (a) Adapted decoder stack. (b) Width-lift and width-return bridges connect paired states to pretrained attention and sparse experts. Attention updates $\bm{a}$ to $\bm{u}$ ; expert processing then updates $\bm{b}$ to $\bm{v}$ before the channel join. (c) Schematic expert bank with frozen router weights during backbone adaptation.

Parameter-efficient adaptation
Method	Memory (GB) $\downarrow$	Samples/s $\uparrow$
LoRA	18.4	76.2
DoRA	19.8	72.5
$(\mathrm{IA})^{3}$	17.6	74.8
Broader weight updates
SFT + checkpointing	64.8	19.9

Table 1: Single-H800 efficiency. Batch sizes are capacity-tuned; throughput is not a matched-token comparison.

Method	MMLU	GSM8K	Multi.	MT-B.
Base model	62.4	61.2	40.4	6.25
LoRA	65.5	71.8	38.7	7.21
DoRA	65.9	71.1	39.1	7.29
$(\mathrm{IA})^{3}$	65.2	70.5	38.4	7.18
SFT + checkpointing	66.3	75.1	39.8	7.56
LOMO	66.5	74.9	39.5	7.52

Table 2: Downstream scores. MMLU, GSM8K, and Multi. are percentages; MT-B. is MT-Bench. Bold marks the best score per column, including the base model.

Figure 2: Memory and MMLU. Points combine the values in Tables 1 and 2 ; they summarize complete configurations with capacity-tuned batches. Throughput and other task scores remain separate criteria.

Configuration	MMLU (%) $\uparrow$
DuplexTune (complete)	67.2
No bridge warm-up (joint updates)	57.5
No backbone adaptation (bridges only)	54.9

Table 3: Training-phase ablation on MMLU. Removing either alignment or backbone fine-tuning reduces the score.

Mixture-of-Experts Fine-Tuning

See all 17 papers

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Jun 15, 2026Martin JaggiMixture-of-Experts Language Models LLM Compression
EPFL
MoRE: Mixture of Reused Experts
Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Department of Computer Science, Cornell University
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google

Jun 15, 2026J/K move · Enter open · S save

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

Martin Jaggi

EPFL

Mixture-of-Experts Language Models LLM Compression LLM Training

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.

Abstract

Mixture-of-Experts Fine-Tuning

Explore similar work

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

MoRE: Mixture of Reused Experts

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models