Complex video editing requires understanding complex instructions while jointly fulfilling multiple editing requests. Meeting these demands through dense model scaling substantially increases computational cost. We introduce VideoEditor-MoE, the first open-source mixture-of-experts framework for video editing. Beyond sparse MLP-MoE layers, MoE-Attention expands attention capacity through routed key--value experts while preserving global interactions among video tokens. Initialized from a 5B video editing model, VideoEditor-MoE scales to 14B and 56B parameters, with approximately 8B parameters activated per token in both variants. Comparisons with existing methods and two complementary baselines, Explicit-MoE and MLP-MoE, assess its benefits over explicit expert guidance and MLP-only expert architectures. Further analyses demonstrate that all experts remain active and reveal that learned routing responds more strongly to sample content and diffusion noise levels than to human-defined editing categories. Spatial analysis further indicates sensitivity to object-level semantic features. We also identify decoupled weight decay as a strategy for stabilizing expert scaling.
Figures & tables
Figure 1: Two MoE baselines. Left: Explicitly Guided MoE sends the complete instruction to the main branch and decomposed instructions to selected experts. Fusion layers injected into the first half of the main branch incorporate expert features. Right: MLP-MoE routes contextualized video tokens to MLP experts while retaining dense attention.
Figure 2: VideoEditor-MoE. (a) Dense and MoE DiT blocks alternate through 28 layers, followed by two dense blocks. (b) Shared queries attend to routed key–value pairs from all video tokens. (c) Each MoE block combines MoE-Attention, dense instruction cross-attention, and MoE-MLP.
Existing methods
MoE baselines
Ours
Metric
Kiwi-Edit
Kiwi-Edit (Finetuned)
MiniMax- H3
CoinVE- Edit
Explicit- MoE
MLP-MoE (19B-A8B)
14B-A8B
56B-A8B
Edit Execution Rate ↑
0.678
0.706
0.736
0.779
0.724
0.810
0.837
0.853
Edit Fulfillment ↑
0.675
0.694
0.727
0.778
0.727
0.813
0.834
0.851
Video Quality ↑
0.592
0.576
0.598
0.677
0.650
0.683
0.704
0.727
FVD/1000 ↓
1.213
1.152
1.493
1.095
1.099
1.268
1.193
1.025
Table 1: Quantitative comparison. VideoEditor-MoE-56B-A8B ranks first and 14B-A8B ranks second in edit execution rate, edit fulfillment, and video quality, while maintaining competitive FVD. Scores are averaged over three benchmarks; per-benchmark scores are reported in Appendix B . Best results are in bold and second-best are underlined . Evaluation protocol appears in Appendix D .
Figure 3: Qualitative comparisons under complex instructions. Each row shows the source video and outputs from five baselines and both VideoEditor-MoE variants. Middle frames are shown, with editing instructions summarized above each row.
Existing methods
MoE baselines
Ours
Cost
Kiwi-Edit
Kiwi-Edit (Finetuned)
MiniMax- H3
CoinVE- Edit
Explicit- MoE
MLP-MoE (19B-A8B)
14B-A8B
56B-A8B
TFLOPs
55
55
2090
2984
260
80
95
95
Time cost (s)
16.4
17.4
596
806
99.2
40.2
16.5
26.1
Table 2: Computational cost. VideoEditor-MoE-56B-A8B requires far fewer TFLOPs and less time than MiniMax-H3 (24B) and CoinVE-Edit (14B). TFLOPs are reported for a single DiT denoising step, and DiT time cost is for generating one video.
Figure 4: Expert utilization and input-dependent allocation. (a) Mean selection shares over 20 samples and three noise levels. Each row sums to 100% , with uniform allocation at approximately 14.3% . (b) Mean Jensen–Shannon (JS) divergence under the comparisons defined in the text. Larger values indicate greater allocation changes.
Figure 5: Object-semantic expert routing. Top: original video. Bottom: routing weights of a representative MLP expert sensitive to this sample’s content. The high routing weight lies in the person region, which demonstrates its sensitivity to semantic objects.
Figure 6: Optimization stability at 56B-A8B. MLP (top) and attention (bottom) outputs under (a) Adam and (b) AdamW. Cells report median relative branch output cℓ over five samples; columns increase input noise level σ . The high outputs of MoE-MLP and MoE-Attention demonstrate that, optimized under AdamW, they exhibit similar computation behavior to dense layers.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training updates
18,630
Clips per GPU / global batch size
8/192
Backbone learning rate
10−5
Expert learning rate
10−4→10−5
Router learning rate
10−3→10−4
Optimizer / weight decay
AdamW / 0.01
Appendix
Table 3: Training configuration for VideoEditor-MoE-14B-A8B. Learning-rate transitions span the first 2,000 updates.
OpenVE-Bench
CoinVE-Bench
CoVE-Bench
Model
Exec.
Fulfill.
Quality
FVD
Exec.
Fulfill.
Quality
FVD
Exec.
Fulfill.
Quality
FVD
Kiwi-Edit
0.769
0.777
0.670
1.039
0.765
0.753
0.620
1.165
0.502
0.495
0.486
1.435
Kiwi-Edit (Finetuned)
0.837
0.828
0.580
1.131
0.731
0.731
0.580
1.129
0.551
0.522
0.569
1.197
MiniMax-H3
0.758
0.748
0.523
1.234
0.870
0.865
0.730
1.609
0.580
0.568
0.542
1.637
CoinVE-Edit
0.875
0.879
0.727
1.008
0.907
0.902
0.720
1.062
0.556
0.553
0.583
1.214
Explicit-MoE
0.856
0.860
0.716
1.008
0.864
0.874
0.720
1.087
0.451
0.446
0.514
1.202
Appendix
Table 4: Per-benchmark quantitative comparison. Exec., Fulfill., and Quality denote Edit Execution Rate, Edit Fulfillment, and Video Quality (higher is better); FVD is reported as FVD/1000 (lower is better). Best results in each column are in bold .
Figure 7: Evaluation criteria and output format for Edit Execution Rate.
Figure 8: Evaluation criteria and output format for Edit Fulfillment.
Figure 9: Evaluation criteria and output format for Video Quality.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Mixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional token-wise MoE routes tokens independently within a homogeneous expert pool and regularizes expert usage toward uniformity, making it poorly matched to video data that is spatiotemporally redundant and semantically long-tailed. We show that existing visual MoEs fall into a uniformity trap: semantically under-organized routing, compounded by uniform expert-usage regularization, scatters coherent patches across disparate experts, causing routing fragmentation and structural distortion. To address this, we propose SplitMoE, a split-role sparse architecture that breaks the shackles of uniformity. To accommodate the inherent semantic imbalance, we explicitly bifurcate the expert pool into semantic experts and generic experts, with semantic experts capturing high-level semantic abstraction and generic experts preserving residual visual information and flexible generative capacity. Leveraging prototype-guided routing and pull-push regularization, SplitMoE enables tokens to cluster naturally by semantic attributes rather than arbitrary balancing constraints. Extensive results show that under an equivalent activated-parameter budget, SplitMoE outperforms traditional load-balanced MoEs in convergence speed, routing coherence, and video generation quality across standard benchmarks. By revealing an emergent coarse-to-fine denoising logic, SplitMoE provides the community with a modality-aware scaling path, serving as a critical reference for building large-scale video world models.
Yu Xu, Yuxin Zhang, Xiao Yang +7
University of Chinese Academy of Sciences · ByteDance · Canva Research +1
We present Mamoda2.5, a unified AR-Diffusion framework that seamlessly integrates multimodal understanding and generation within a single architecture. To efficiently enhance the model's generation capability, we equip the Diffusion Transformer backbone with a fine-grained Mixture-of-Experts (MoE) design (128 experts, Top-8 routing), yielding a 25B-parameter model that activates only 3B parameters, significantly reducing training costs while scaling up the model capacity. Mamoda2.5 achieves top-tier generation performance on VBench 2.0 and sets a new record in video editing quality, surpassing evaluated open-source models and matching the performance of current top-tier proprietary models, including the Kling O1 on OpenVE-Bench. Furthermore, we introduce a joint few-step distillation and reinforcement learning framework that compresses the 30-step editing model into a 4-step model and greatly accelerates model inference. Compared to open-source baselines, Mamoda2.5 achieves up to 95.9× faster video editing inference. In real-world applications, Mamoda2.5 has been successfully deployed for content moderation and creative restoration tasks in advertising scenarios, achieving a 98% success rate in internal advertising video editing scenario.