VideoEditor-MoE: Scaling Complex Video Editing with Mixture of Experts
Organizations: Tencent · National University of Singapore · Xiamen University
Abstract
Complex video editing requires understanding complex instructions while jointly fulfilling multiple editing requests. Meeting these demands through dense model scaling substantially increases computational cost. We introduce VideoEditor-MoE, the first open-source mixture-of-experts framework for video editing. Beyond sparse MLP-MoE layers, MoE-Attention expands attention capacity through routed key--value experts while preserving global interactions among video tokens. Initialized from a 5B video editing model, VideoEditor-MoE scales to 14B and 56B parameters, with approximately 8B parameters activated per token in both variants. Comparisons with existing methods and two complementary baselines, Explicit-MoE and MLP-MoE, assess its benefits over explicit expert guidance and MLP-only expert architectures. Further analyses demonstrate that all experts remain active and reveal that learned routing responds more strongly to sample content and diffusion noise levels than to human-defined editing categories. Spatial analysis further indicates sensitivity to object-level semantic features. We also identify decoupled weight decay as a strategy for stabilizing expert scaling.
Figures & tables
| Existing methods | MoE baselines | Ours | ||||||
|---|---|---|---|---|---|---|---|---|
| Metric | Kiwi-Edit | Kiwi-Edit (Finetuned) | MiniMax- H3 | CoinVE- Edit | Explicit- MoE | MLP-MoE (19B-A8B) | 14B-A8B | 56B-A8B |
| Edit Execution Rate | 0.678 | 0.706 | 0.736 | 0.779 | 0.724 | 0.810 | 0.837 | 0.853 |
| Edit Fulfillment | 0.675 | 0.694 | 0.727 | 0.778 | 0.727 | 0.813 | 0.834 | 0.851 |
| Video Quality | 0.592 | 0.576 | 0.598 | 0.677 | 0.650 | 0.683 | 0.704 | 0.727 |
| FVD/1000 | 1.213 | 1.152 | 1.493 | 1.095 | 1.099 | 1.268 | 1.193 | 1.025 |
| Existing methods | MoE baselines | Ours | ||||||
|---|---|---|---|---|---|---|---|---|
| Cost | Kiwi-Edit | Kiwi-Edit (Finetuned) | MiniMax- H3 | CoinVE- Edit | Explicit- MoE | MLP-MoE (19B-A8B) | 14B-A8B | 56B-A8B |
| TFLOPs | 55 | 55 | 2090 | 2984 | 260 | 80 | 95 | 95 |
| Time cost (s) | 16.4 | 17.4 | 596 | 806 | 99.2 | 40.2 | 16.5 | 26.1 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Training updates | 18,630 |
| Clips per GPU / global batch size | |
| Backbone learning rate | |
| Expert learning rate | |
| Router learning rate | |
| Optimizer / weight decay | AdamW / 0.01 |
| OpenVE-Bench | CoinVE-Bench | CoVE-Bench | ||||||||||
| Model | Exec. | Fulfill. | Quality | FVD | Exec. | Fulfill. | Quality | FVD | Exec. | Fulfill. | Quality | FVD |
| Kiwi-Edit | 0.769 | 0.777 | 0.670 | 1.039 | 0.765 | 0.753 | 0.620 | 1.165 | 0.502 | 0.495 | 0.486 | 1.435 |
| Kiwi-Edit (Finetuned) | 0.837 | 0.828 | 0.580 | 1.131 | 0.731 | 0.731 | 0.580 | 1.129 | 0.551 | 0.522 | 0.569 | 1.197 |
| MiniMax-H3 | 0.758 | 0.748 | 0.523 | 1.234 | 0.870 | 0.865 | 0.730 | 1.609 | 0.580 | 0.568 | 0.542 | 1.637 |
| CoinVE-Edit | 0.875 | 0.879 | 0.727 | 1.008 | 0.907 | 0.902 | 0.720 | 1.062 | 0.556 | 0.553 | 0.583 | 1.214 |
| Explicit-MoE | 0.856 | 0.860 | 0.716 | 1.008 | 0.864 | 0.874 | 0.720 | 1.087 | 0.451 | 0.446 | 0.514 | 1.202 |
Mixture-of-Experts Models
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
See all 484 papers