Communicate the Change: Efficient Expert Parallelism via Activation Compression
Organizations: Mila - Quebec AI Institute · McGill University · Canada CIFAR AI Chair · Periodic Labs
Abstract
Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.
Figures & tables
| Architecture | Inter-node RDMA Communication | ||||
| Model | MiB/token | GiB/128K | Transfer (ms) | ||
| Qwen3-30B-A3B | 2048 | 48 | 1.125 | 144 | 0 97 |
| Qwen3-235B-A22B | 4096 | 94 | 4.406 | 564 | 379 |
| DeepSeek-V3 | 7168 | 58 | 4.758 | 609 | 409 |
| Model | Cosine similarity (%) | Predictability (%) | |||||||
| Qwen3-235B-A22B | 94 | 94.9 | 82.8 | 70.9 | 56.6 | 90.2 | 69.4 | 52.1 | 35.1 |
| DeepSeek-V3 | 58 | 93.7 | 77.5 | 61.5 | 39.7 | 88.1 | 62.2 | 40.3 | 17.5 |
| GLM-5.3 | 75 | 93.4 | 76.7 | 60.0 | 38.0 | 87.3 | 60.1 | 38.2 | 16.6 |
| Qwen3-30B-A3B | 48 | 91.6 | 74.9 | 62.0 | 47.0 | 83.2 | 56.0 | 40.1 | 25.1 |
| Kanana-2-30B-A3B | 47 | 90.9 | 68.5 | 46.7 | 22.5 | 82.9 | 52.0 | 28.9 | 8.9 |
| Configuration | Task Performance (%) | |||||||
| Stride | Dispatch | Combine | RDMA Reduction | Knowledge & Reasoning | Math | Code | Instruction Following | Overall ( ) |
| Qwen3-30B-A3B | ||||||||
| Uncompressed ( ) | 77.6 | 50.7 | 59.2 | 55.6 | 66.9 | |||
| 4 | 1280 | 1280 | 76.2 | 49.1 | 58.1 | 54.5 | 65.5 ( 1.4) | |
| 4 | 512 | 1280 | 75.3 | 49.3 | 56.8 | 53.7 | 64.9 ( 2.0) | |
| 4 | 512 | 1024 | 74.3 | 47.8 | 55.2 | 53.5 | 63.8 ( 3.1) | |
Appendix figures & tables57 assets
Supplementary material from the paper’s appendix.
Appendix
| Code width | Own energy in top | Input energy dropped | ||
| Raw input | Residual | Plain code | Residual code | |
| 128 | ||||
| 256 | ||||
| 512 | ||||
| 1024 | ||||
| Setting | Value |
| Loss function | Cross-entropy |
| Training epochs | |
| Training steps | |
| Global batch size | packed sequences |
| Sequence length | tokens |
| Training tokens | B |
| Compression setting | Dev mode | Adaptation |
| Dispatch-only | – | |
| Combine-only | – | |
| Joint |
| Setting | Dev mode | Adaptation |
| Trainable parameters | Compression parameters | All model parameters |
| Folding | After training | Before training |
| Training steps | ||
| Warmup steps | ||
| Training tokens | B | B |
| Dolmino | M | B |
| Setting | Value |
| Global batch size | packed sequences |
| Sequence length | tokens |
| Distillation temperature | |
| KL loss weight | |
| Cross-entropy loss weight | |
| Optimizer | Adam |
| Stage | Tokens | H100 GPU-hours per run |
| Full-model adaptation | B | |
| Dev mode (configuration search) | B | |
| Initialization statistics | K positions |
| Configuration | |||||
| Stride | Dispatch | Combine | Throughput speedup | GPU-hours saved per hour | Break-even |
| 4 | 512 | 1280 | h ( days) | ||
| 4 | 512 | 1024 | h ( days) | ||
| 4 | 512 | 768 | h ( days) | ||
| 8 | 256 | 512 | h ( days) | ||
| Benchmark | Non-chat prompt | Chat prompt |
| Knowledge and reasoning | ||
| MMLU ( Hendrycks et al., 2021 ) | 5-shot | 0-shot, CoT |
| MMLU-Pro ( Wang et al., 2024 ) | 5-shot, CoT | 0-shot, CoT |
| MMLU-Redux ( Gema et al., 2025 ) | 5-shot | 0-shot, CoT |
| ARC-Easy ( Clark et al., 2018 ) | 5-shot | 0-shot, CoT |
| ARC-Challenge ( Clark et al., 2018 ) | 5-shot | 0-shot, CoT |
| Plain compression | EchoMoE | ||||||||
| Configuration | Configuration | ||||||||
| Dispatch | Combine | RDMA Reduction | Overall (%) | Stride | Dispatch | Combine | RDMA Reduction | Overall (%) | (pp) |
| Dev mode | |||||||||
| 1280 | 1280 | 61.9 | 4 | 512 | 1280 | 63.4 | |||
| 1280 | 1024 | 60.2 | 4 | 512 | 1280 | 63.4 | |||
| 1024 | 1024 | 56.9 | 4 | 512 | 1024 | 62.0 | |||
| Configuration | RDMA Reduction | Overall (%) | Matrix MACs |
| Qwen3-30B-A3B ( ) | 66.9 | ||
| About | |||
| EchoMoE | 64.9 ( 2.0) | ||
| Plain | 62.8 ( 4.1) | ||
| About | |||
| EchoMoE | 63.8 ( 3.1) | ||
| Experts shared (%) | Router KL | ||||||
| Initialization | After adaptation | After adaptation | |||||
| Configuration | RDMA reduction | Dolci | Dolmino | Dolci | Dolmino | Dolci | Dolmino |
| EchoMoE | |||||||
| EchoMoE | |||||||
| Plain | |||||||
| EchoMoE | |||||||
| Task Performance (%) | ||||||
| Method | RDMA Reduction | Knowledge & Reasoning | Math | Code | Instruction Following | Overall ( ) |
| Uncompressed | 77.6 | 50.7 | 59.2 | 55.5 | 66.9 | |
| EchoMoE | 74.3 | 47.8 | 55.2 | 53.4 | 63.8 ( 3.1) | |
| LSH-MoE | 65.3 | 44.2 | 53.1 | 46.7 | 57.1 ( 9.8) | |
| Predicted from | Qwen3-30B-A3B | Qwen3-235B-A22B | DeepSeek-V3 |
| Same token, layers earlier (EchoMoE) | |||
| Best other token, same expert | |||
| LSH-MoE representative, six hashes |
| Overall drop (pp) | |||||
| Dispatch | Reference stride | Plain dispatch | |||
| 4 | 6 | 8 | 12 | ||
| 1280 | 0.0 | 1.1 | 1.0 | 1.6 | – |
| 1024 | 0.6 | 1.8 | 2.3 | 3.0 | 7.1 |
| 768 | 1.0 | 2.5 | 3.9 | 5.3 | 12.4 |
| 512 | 2.6 | 5.5 | 6.3 | 10.6 | 21.1 |
| Configuration | |||||
| Stride | Dispatch | Combine | RDMA Reduction | Overall (%) | (pp) |
| 8 | 768 | 1280 | 62.1 | – | |
| 4 | 512 | 1280 | 63.4 | ||
| 8 | 768 | 1024 | 60.0 | – | |
| 4 | 512 | 1024 | 62.0 | ||
| 8 | 768 | 768 | 57.0 | – | |
| Configuration | Direct decoding | Change from direct decoding | |
| Combine | Overall (%) | Bias (pp) | Gated bias (pp) |
| Qwen3-30B-A3B ( ) | 66.9 | – | – |
| 1280 | 65.2 | ||
| 1024 | 64.1 | ||
| 768 | 60.6 | ||
| 512 | 54.7 | ||
| Configuration | BF16 RDMA Reduction | |||||
| Stride | Dispatch | Combine | EP16 nodes | EP32 nodes | EP64 nodes | EP128 nodes |
| Qwen3-30B-A3B ( , experts) | ||||||
| 4 | 1280 | 1280 | ||||
| 4 | 512 | 1280 | ||||
| 4 | 512 | 1024 | ||||
| 4 | 512 | 768 | ||||
| Configuration | Overall (%) | ||||
| Stride | Dispatch | Combine | BF16 | FP8 | (pp) |
| Qwen3-30B-A3B ( , experts) | 66.63 | 66.83 | |||
| EchoMoE | |||||
| 4 | 1280 | 1280 | 65.54 | 65.41 | |
| 6 | 1280 | 1280 | 65.13 | 65.17 | |
| 12 | 1280 | 1280 | 64.11 | 63.92 | |
| Configuration | Relative Matrix MACs | ||||||
| Stride | Dispatch | Combine | BF16 RDMA Reduction | Parameters (B) | Expert | Owner | Total |
| Qwen3-30B-A3B ( , experts) | 30.53 | – | |||||
| 4 | 512 | 1280 | 30.70 | ||||
| 4 | 512 | 768 | 28.23 | ||||
| 6 | 256 | 768 | 26.60 | ||||
| 8 | 256 | 512 | 25.47 | ||||
| Setting | Value |
| Prefill scheduling | Chunked prefill |
| Token budget per step | |
| Activation transfers | BF16 |
| Collectives | DeepEP V2 hybrid mode or two-hop kernels, per call |
| Expert assignment | Balanced |
| DP synchronization | NCCL |
| Component | Role in EchoMoE |
| Existing components | |
| Expert kernels (fused MoE GEMMs) | Run the folded weights as ordinary expert weights with different input and output widths |
| Dispatch and combine kernels (DeepEP) | Carry the narrower rows, with an optional hybrid-mode connection patch that the uncompressed model also uses |
| Reference collectives (NCCL) | Broadcast full-width states at reference layers |
| Added components | |
| Owner-rank compression kernels | Encode dispatch codes and decode combine codes |
| Configuration | Input / output tokens (requests per GPU) | ||||||
| Stride | Dispatch | Combine | 1024/128 (512) | 4096/256 (128) | 16384/512 (32) | 128000/512 (4) | 128/1024 (512) |
| Throughput speedup ( ) | |||||||
| 4 | 1280 | 1280 | |||||
| 6 | 1280 | 1280 | |||||
| 4 | 512 | 1280 | |||||
| 4 | 512 | 1024 | |||||
| Workload | Requests per GPU | ||||||
| 128/1024 | |||||||
| 1024/128 | |||||||
| 4096/256 | – | – | |||||
| 16384/512 | – | – | – | – | |||
| Configuration | 4096/256 ( requests per GPU) | 128000/512 ( requests per GPU) | ||
| Throughput | TTFT | Throughput | TTFT | |
| Configuration | Input / output tokens | |||
| 128/1024 | 1024/128 | 4096/256 | 16384/512 | |
| – | ||||
| Configuration ( ) | Prefill (ms) | Decode (ms) | ||||
| Saved | Reference | Saved | Reference | |||
| 128/1024, requests per GPU | ||||||
| 4096/256, requests per GPU | ||||||
| Configuration ( ) | Throughput speedup ( ) | ||
| Overall (%) | vs. FP8 | vs. BF16 | |
| Qwen3-30B-A3B ( , experts) | 66.83 | ||
| 64.86 ( 2.0) | |||
| 62.99 ( 3.8) | |||
| 55.56 ( 11.3) | |||
| Input / output tokens (requests per GPU) | ||||||
| Configuration | 1024/128 (512) | 4096/256 (128) | 16384/512 (32) | 128000/512 (4) | 128/1024 (512) | 4096/256 (8) |
| Throughput speedup ( ) | ||||||
| TTFT speedup ( ) | ||||||
| Configuration ( ) | KV cache | |||
| Model load (GiB) | Memory (GiB) | Tokens | Change | |
| Qwen3-30B-A3B ( , experts) | – | |||
| Configuration | Throughput speedup ( ) | |
| InfiniBand | TCP | |
| Throughput speedup ( ) | |||
| Network | Bandwidth | BF16 | FP8 |
| InfiniBand | Gb/s per NIC | – | – |
| InfiniBand, capped | Gb/s per NIC | – | – |
| InfiniBand, capped | Gb/s per NIC | – | – |
| TCP over Ethernet | Gb/s per node | – | – |
| Uncompressed / EchoMoE | ||||
| Requests/s | P99 TTFT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Output (k tokens/s) |
| BF16, real routing | ||||
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / | |
| BF16 communication | FP8 communication | |||||
| Configuration | Throughput | TTFT | TPOT | Throughput | TTFT | TPOT |
| Agent turn: K/ with requests per GPU | ||||||
| Agent turn: K/ with requests per GPU | ||||||
| Code width | ||||
| Model | Dispatch | Combine | Expert weights | KV cache per GPU |
| Qwen3-235B-A22B | BF16 | GiB | ||
| DeepSeek-V3 | FP8 | GiB | ||
| Kimi-K2.6 | INT4 | GiB | ||
| BF16 communication | FP8 communication | |||||
| Model (RDMA reduction) | Uncompressed | EchoMoE | Speedup | Uncompressed | EchoMoE | Speedup |
| Qwen3-235B-A22B ( ) | ||||||
| DeepSeek-V3 ( ) | ||||||
| Kimi-K2.6 ( ) | ||||||
| BF16 communication | FP8 communication | |||||
| Workload (requests per GPU) | Throughput | TTFT | TPOT | Throughput | TTFT | TPOT |
| Kanana-2-30B-A3B, | ||||||
| 1024/128 ( ) | ||||||
| 4096/256 ( ) | ||||||
| 16384/512 ( ) | ||||||
| 128/1024 ( ) | ||||||
| Qwen3 | Gemma 4 | Kanana-2 | Nemotron 3.5 | Moonlight | |||||||
| Benchmark | Prompt | Unc. | Unc. | Unc. | Unc. | Unc. | |||||
| Knowledge and reasoning | |||||||||||
| MMLU | Few-shot | ||||||||||
| ARC-Easy | Few-shot | ||||||||||
| ARC-Challenge | Few-shot | ||||||||||
| HellaSwag | Few-shot | ||||||||||
| Configuration | Task Performance (%) | |||||||
|---|---|---|---|---|---|---|---|---|
| Stride | Dispatch | Combine | RDMA Reduction | Knowledge & Reasoning | Math | Code | Instruction Following | Overall ( ) |
| Qwen3-30B-A3B ( ) | 77.6 | 50.7 | 59.2 | 55.6 | 66.9 | |||
| EchoMoE | ||||||||
| 4 | 1280 | 1280 | 75.8 | 50.0 | 56.9 | 53.8 | 65.4 ( 1.5) | |
| 6 | 1280 | 1280 | 75.5 | 49.3 | 55.6 | 53.2 | 64.8 ( 2.1) | |
| 8 | 1280 | 1280 | 74.8 | 49.6 | 56.0 | 54.6 | 64.7 ( 2.2) | |
| Configuration | Task Performance (%) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Stride | Dispatch | Combine | RDMA Reduction | Knowledge & Reasoning | Math | Code | Instruction Following | Overall ( ) | (pp) |
| Qwen3-30B-A3B ( ) | 77.6 | 50.7 | 59.2 | 55.6 | 66.9 | – | |||
| EchoMoE | |||||||||
| 4 | 1280 | 1280 | 76.2 | 49.1 | 58.1 | 54.5 | 65.5 ( 1.4) | ||
| 6 | 1280 | 1280 | 75.6 | 49.1 | 57.4 | 53.8 | 65.1 ( 1.8) | ||
| 12 | 1280 | 1280 | 73.7 | 49.6 | 57.4 | 53.1 | 64.1 ( 2.8) | ||