cs.CVAug 25, 2026
SaveMoTE: Mixture of Task Experts for Multi-Task Video Understanding
Organizations: Department of Computer Science University of Kaiserslautern-Landau (RPTU) Kaiserslautern, Germany · Augmented Vision Group German Research Center for Artificial Intelligence (DFKI) Kaiserslautern, Germany
Abstract
Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction. Dense transformer decoders share the same feed-forward networks across tasks, which can entangle task behavior and make controlled capability expansion difficult. Sparse Mixture-of-Experts (MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We propose MoTE (Mixture of Task Experts), a decoder architecture that converts large language model feed-forward networks into task-specific experts while keeping the multimodal backbone shared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design as VideoLLM-MoTE and evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.
Figures & tables
Figure 1: VideoLLM-MoTE overview. A user prompt and sampled video frames are converted into prompt and visual tokens before entering an LLM decoder modified with MoTE. Prompt-conditioned expert selection matches the prompt to registered expert descriptions and passes the selected expert id to the decoder. Given the input tokens and expert id, LLM generates a response. The visual encoder and base language model are non-trainable, while the MLP projector, LoRA adapters, and selected task experts are trainable.
| Method | Domain | Routing Level | Routing Type | Interpretability | Task-Aware |
|---|---|---|---|---|---|
| Sparse MoE [ Shazeer et al.(2017)Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton, and Dean ] | Language | Token | Learned top- | Low | No |
| GShard [ Lepikhin et al.(2021)Lepikhin, Lee, Xu, Chen, Firat, Huang, Krikun, Shazeer, and Chen ] | Language | Token | Learned top- | Low | No |
| Switch Transformer [ Fedus et al.(2022)Fedus, Zoph, and Shazeer ] | Language | Token | Learned top-1 | Low | No |
| V-MoE [ Riquelme et al.(2021)Riquelme, Puigcerver, Mustafa, Neumann, Jenatton, Susano Pinto, Keysers, and Houlsby ] | Vision | Patch | Learned top- | Low | No |
| Sparse Upcycling [ Komatsuzaki et al.(2023)Komatsuzaki, Puigcerver, Lee-Thorp, Ruiz, Mustafa, Ainslie, Tay, Dehghani, and Houlsby ] | Language/vision | Token | Expert Choice/Learned top- | Low | No |
| LLaMA-MoE [ Zhu et al.(2024)Zhu, Qu, Dong, Ruan, Tong, He, and Cheng ] | Language | Token | Learned top- | Low | No |
Table 1: Comparison of MoE approaches. Prior sparse transformers primarily route tokens or patches with learned gates. Task-level methods make routing more aligned with task semantics by using task-conditioned features in the gating function, but they are not designed for modular video-language decoders. MoTE uses sample-level task routing over decoder FFN experts; the reported experiments use explicit task routes, while prompt-description matching provides an expert-selection component for inference.
Figure 2: MoTE decoder and expert selection. (a) In each decoder layer, attention remains shared, while the FFN layer is replaced by a shared FFN expert and a switch-selected task FFN experts. Both expert outputs are added to the residual stream. (b) Task experts are initialized by copying the pretrained dense FFN into multiple task branches. (c) Prompt-conditioned expert selection embeds the prompt and registered expert descriptions, computes cosine similarities, and selects the top-1 expert for the full video–prompt sample.
| Method | Not use HowTo100M | COIN Benchmark Top-1 Accuracy | Avg. | ||||
|---|---|---|---|---|---|---|---|
| Step | Next | Proc. | Task | Proc.+ | |||
| \textcolor gray Non-LLM procedural video methods | |||||||
| ClipBERT [ Lei et al.(2021)Lei, Li, Zhou, Gan, Berg, Bansal, and Liu ] | \textcolor tickgreen✓ | 30.8 | – | – | 65.4 | – | – |
| TimeSformer [ Bertasius et al.(2021)Bertasius, Wang, and Torresani ] | \textcolor crossred✗ | 46.5 | 34.0 | 17.0 | 85.3 | 40.1 | 44.6 |
| Paprika [ Zhou et al.(2023)Zhou, Martín-Martín, Kapadia, Savarese, and Niebles ] | \textcolor crossred✗ | 51.0 | 43.2 | – | 85.8 | – | – |
| DistantSup [ Lin et al.(2022)Lin, Petroni, Bertasius, Rohrbach, Chang, and Torresani ] | \textcolor crossred✗ | 54.1 | 39.4 | – | 90.0 | 41.3 | – |
Table 2: Comparison with prior COIN baselines. Not use HowTo100M indicates whether the method avoids HowTo100M pretraining. The average is reported when all five task scores are available. Our method achieves 62.9% average accuracy outperforming other baselines.
| Model | LLM | Frame | Efficiency | Params (B) | COIN Avg. | |||
|---|---|---|---|---|---|---|---|---|
| Strategy | TFLOPs | Latency | Dec/s | Total | Active | |||
| VideoLLM-online-7B-v1 [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | Llama-2-7B | 1 | 9.8 | 2.3 | 28.6 | 6.7 | 6.7 | 60.2 |
| VideoLLM-online-8B-v1+ [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | Llama-3-8B | 1+3x3 | 98.7 | 2.8 | 27.8 | 8.0 | 8.0 | 61.8 |
| VideoLLM-MoD [ Wu et al.(2024)Wu, Chen, Lin, Wang, Gao, Xu, Xu, Hu, Chen, and Shou ] | Llama-3-8B | 1+3x3 | 60.0 | 3.0 | 24.6 | 8.0 | 8.0 | 61.8 |
| VideoLLM-online-1B-v1+ | Llama-3.2-1B | 1+3x3 | 15.2 | 1.3 | 54.7 | 1.2 | 1.2 | 59.3 |
| \rowcolor oursrow VideoLLM-MoTE-1B+5E (ours) | Llama-3.2-1B | 1+3x3 | 28.9 | 1.8 | 37.7 | 5.3 | 2.0 | 62.9 |
Table 3: Efficiency, LLM-only parameter summary and COIN average accuracy. Profiling uses one NVIDIA A100 with 600 frames, 64 prompt tokens, and 64 generated tokens. Parameter counts include token embeddings and transformer decoder layers, but exclude visual encoders and MLP projectors. VideoLLM-MoTE-1B+5E achieves the highest COIN average while activating only 2B LLM parameters per sample.
| COIN | Stage 1 | Expert addition stages | Expert removal stages | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Benchmark | Step | \textcolor tickgreen ✚ Next | \textcolor tickgreen ✚ Proc. | \textcolor tickgreen ✚ Task | \textcolor tickgreen ✚ Proc.+ | \textcolor crossred ✖ Proc.+ | \textcolor crossred ✖ Task | \textcolor crossred ✖ Proc. | \textcolor crossred ✖ Next |
| Step | 63.1 | 63.1 | 63.1 | 63.1 | 63.1 | 63.1 | 63.1 | 63.1 | 63.1 |
| Next | – | 46.2 | 46.2 | 46.2 | 46.2 | 46.2 | 46.2 | 46.2 | – |
| Proc. | – | – | 48.4 | 48.4 | 48.4 | 48.4 | 48.4 | – | – |
| Task | – | – | – | 92.8 | 92.8 | 92.8 | – | – | – |
| Proc.+ | – | – | – | – | 54.6 | – | – | – | – |
Table 4: Modular task adaptation through sequential expert addition and removal . We first train a single Step expert, then add one task expert at a time for Next , Proc. , Task , and Proc.+ . After all five experts are available, we remove them in reverse order. Accuracy is reported after each addition or removal stage, and a dash indicates that the corresponding task expert is not present. Stable scores for previously added tasks indicate that adding or removing other experts does not alter their routed inference paths.
| Method | Model instance | COIN Benchmark Top-1 Accuracy | Ego4D Narration Stream Validation | ||||||
| Step | Next | Proc. | Task | Proc.+ | PPL | TimeDiff | Fluency | ||
| Separate finetuned models reported by prior work | |||||||||
| VideoLLM-online-7B-v1 [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | COIN-only | 59.8 | 48.1 | 47.9 | 92.1 | 52.9 | – | – | – |
| VideoLLM-online-7B-v1 [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | Ego4D-only | – | – | – | – | – | 2.43 | 2.25 | 42.6 |
| VideoLLM-online-8B-v1+ [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | COIN-only | 63.1 | 49.1 | 49.8 | 92.7 | 54.1 | – | – | – |
| VideoLLM-online-8B-v1+ [ Chen et al.(2024)Chen, Lv, Wu, Lin, Song, Gao, Liu, Gao, Mao, and Shou ] | Ego4D-only | – | – | – | – | – | 2.40 | 2.05 | 45.3 |
Table 5: Cross-dataset task extension from online narration to offline procedural tasks. Prior VideoLLM-online and VideoLLM-MoD results are reported from separate COIN-only and Ego4D-only finetuned model instances. In contrast, the final row evaluates one expanded VideoLLM-MoTE-1B+6E checkpoint that starts from an Ego4D narration route and adds five COIN task routes. Ego4D validation reports PPL, TimeDiff, and Fluency.
| Method | COIN Benchmark Top-1 Accuracy | Avg. | ||||
|---|---|---|---|---|---|---|
| Step | Next | Proc. | Task | Proc.+ | ||
| VideoLLM-1B+5E-Dense | 64.4 \textcolor promptlabel (0.0) | 49.8 \textcolor promptlabel (0.0) | 48.7 \textcolor promptlabel (0.0) | 93.2 \textcolor promptlabel (0.0) | 51.5 \textcolor promptlabel (0.0) | 61.5 \textcolor promptlabel (0.0) |
| VideoLLM-MoE-Top1-token | 63.7 \textcolor crossred (-0.7) | 49.5 \textcolor crossred (-0.3) | 48.5 \textcolor crossred (-0.2) | 93.8 \textcolor tickgreen (+0.6) | 51.7 \textcolor tickgreen (+0.2) | 61.4 \textcolor crossred (-0.1) |
| VideoLLM-MoE-Top1-sample | 64.9 \textcolor tickgreen (+0.5) | 50.3 \textcolor tickgreen (+0.5) | 49.2 \textcolor tickgreen (+0.5) | 93.6 \textcolor tickgreen (+0.4) | 52.3 \textcolor tickgreen (+0.8) | 62.1 \textcolor tickgreen (+0.6) |
| \rowcolor oursrow VideoLLM-MoTE-1B+5E (ours) | 65.1 \textcolor tickgreen (+0.7) | 50.3 \textcolor tickgreen (+0.5) | 50.5 \textcolor tickgreen (+1.8) | 94.4 \textcolor tickgreen (+1.2) | 54.2 \textcolor tickgreen (+2.7) | 62.9 \textcolor tickgreen (+1.4) |
Table 8: Effect of Routing Granularity . Numbers in parentheses are accuracy changes relative to the VideoLLM-1B+5E-Dense control; green and red indicate positive and negative changes, respectively. VideoLLM-MoE-Top1-token learns token-level routes, VideoLLM-MoE-Top1-sample learns one route for the full sample, and VideoLLM-MoTE-1B+5E uses explicit task-index routing.
| Method | MoE Construction | COIN Benchmark Top-1 Accuracy | Avg. | ||||
|---|---|---|---|---|---|---|---|
| Step | Next | Proc. | Task | Proc.+ | |||
| VideoLLM-MoTE-8B+5E-Split | Splitting FFN | 63.3 | 48.8 | 48.3 | 92.9 | 48.3 | 60.3 |
| \rowcolor oursrowVideoLLM-MoTE-1B+5E (default) | Copying FFN | 65.1 | 50.3 | 50.5 | 94.4 | 54.2 | 62.9 |
Table 9: Effect of Expert Construction Method . VideoLLM-MoTE-8B-5E-Split partitions each VideoLLM FFN into six equally sized expert groups (one shared expert and five task experts), while VideoLLM-MoTE-1B+5E initializes each shared and task expert as a full copied FFN. Copying FFN layer into multiple experts achieves better performance.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyper-parameter | Value |
|---|---|
| frame_fps / max_frames | 2 / 1200 |
| GPUs | 4 A100-80GB |
| per_device_batch | 1 train / 1 eval |
| gradient_acc_steps | 8 |
| Effective train batch | 32 samples |
| num_train_epochs | 10 |
Table 10: Hyper-parameters used for training VideoLLM-MoTE.
Figure 3: Bucketed rank-aligned sampling. The sampler groups a concatenated multi-task dataset into expert buckets. At each local training step, it samples one expert bucket with probability proportional to the number of fresh samples remaining in that bucket, draws one global group of indices from the selected bucket, and lets distributed sharding split the group into rank-local microbatches. Here is the local batch size and is the number of data-parallel ranks.
| Task | Expert natural-language description |
|---|---|
| Current Step | Recognize the action currently being performed in the observed video segment; use this route for prompts asking what is happening now or which procedural step is visible. |
| Next Step | Predict the immediate next action that should follow the observed segment; use this route for short-horizon forecasting prompts asking what comes next. |
| Task | Identify the overall instructional goal or activity category of the video, independent of the current substep; use this route for prompts asking for the complete task being performed. |
| Procedure | Forecast a sequence of upcoming procedural actions from the current visual state without being given an explicit task goal; use this route for open-ended procedure-continuation prompts. |
| Task Procedure | Forecast upcoming procedural actions while conditioning on a provided goal or task name; use this route when the prompt states the target activity and asks for future steps. |
| Narration | Generate concise timestamp-aligned egocentric narrations for streaming video; use this route for online narration prompts that require deciding both what to say and when to say it. |
Table 11: Task-to-description mapping used for prompt-conditioned expert selection.
| Task route | Representative generated prompt variants |
|---|---|
| \rowcolor promptbgStep | \textcolor promptlabel 1. Which action is visible in this clip? Use a minimal answer with no extra text. \textcolor promptlabel 2. Recognize the step currently shown in the video. Use a minimal answer with no extra text. \textcolor promptlabel 3. What action does this video segment depict? Answer briefly with only the action name. \textcolor promptlabel … 47 additional variants |
| Next | \textcolor promptlabel 1. Name the step that follows the current video. Give just the next action label. \textcolor promptlabel 2. What should the person do next? Keep it short and direct. \textcolor promptlabel 3. What is the upcoming action after the video segment? Keep it short and direct. \textcolor promptlabel … 47 additional variants |
| \rowcolor promptbgTask | \textcolor promptlabel 1. What larger task does this video demonstrate? Give just the overall task label. \textcolor promptlabel 2. What is the person trying to accomplish overall? Return a short activity phrase only. \textcolor promptlabel 3. State the overall procedure represented by the clip. Answer concisely with only the task name. \textcolor promptlabel … 47 additional variants |
| Proc. | \textcolor promptlabel 1. Name the upcoming {num_steps} actions in order. Return only the ordered action list. \textcolor promptlabel 2. What sequence of {num_steps} actions follows this segment? Keep the list concise and include no commentary. \textcolor promptlabel 3. Determine the next {num_steps} steps after the shown action. Keep the list concise and include no commentary. \textcolor promptlabel … 47 additional variants |
| \rowcolor promptbgProc.+ | \textcolor promptlabel 1. Using {task} as the overall activity, name the upcoming {num_steps} actions. Do not include any extra text beyond the answer. \textcolor promptlabel 2. For {task} , identify the next action based on the current video. Return only the action or numbered action list. \textcolor promptlabel 3. In the context of {task} , what step comes after the shown segment? Use numbered lines when multiple actions are requested. \textcolor promptlabel … 47 additional variants |
Table 12: Representative generated prompt variants used to evaluate prompt-conditioned expert selection. Three of 50 variants are shown per task; the held-out split achieves the 100% test F1 reported in the text.
| Method | COIN Benchmark Top-1 Accuracy | ||||
|---|---|---|---|---|---|
| Step | Next | Proc. | Task | Proc.+ | |
| \rowcolor oursrowVideoLLM-MoTE-1B+5E | 65.1 | 50.3 | 50.5 | 94.4 | 54.2 |
| Swap Step Next | \cellcolor swapcell21.5 | \cellcolor swapcell17.4 | 50.5 | 94.4 | 54.2 |
| Swap Next Proc. | 65.1 | \cellcolor swapcell23.3 | \cellcolor swapcell30.7 | 94.4 | 54.2 |
| Swap Proc. Task | 65.1 | 50.3 | \cellcolor swapcell4.97 | \cellcolor swapcell15.36 | 54.2 |
| Swap Task Proc.+ | 65.1 | 50.3 | 50.5 | \cellcolor swapcell15.98 | \cellcolor swapcell5.36 |
Table 13: Offline expert-swap routing diagnostic on COIN. Highlighted cells indicate the two task columns whose expert weights are swapped at evaluation time. Untouched task routes keep the base route weights.
Figure 4: GLM-OCR-MoTE document processing architecture. Detected document regions pass through the shared CogVIT encoder and GLM-OCR attention path before the switch activates one task-specific MLP expert alongside the shared expert. Solid lines denote active routes and dotted lines denote inactive routes.
| Method | Input Modality | SROIE Task 3 Strict F1 (%) (KIE Domain) | OmniDocBench v1.5 (Original Task) | |||||||||
| Comp. | Date | Addr. | Total | Micro | Over. | TxtEd. | Form. | Tab. | Tab-S | RdEd. | ||
| Text-Only Baselines (OCR Prompted) | ||||||||||||
| GPT-4 + OCR [ OpenAI(2023) ] | OCR Text Only | – | – | – | – | 90.60 | – | – | – | – | – | – |
| Llama2-7B-chat [ Touvron et al.(2023) ] | OCR Text Only | – | – | – | – | 68.97 | – | – | – | – | – | – |
| Vicuna-1.5-7B [ Chiang et al.(2023) ] | OCR Text Only | – | – | – | – | 68.20 | – | – | – | – | – | – |
| Llama2 + OCR [ Touvron et al.(2023) ] | OCR Text Only | – | – | – | – | 56.40 | – | – | – | – | – | – |
Table 14: Full SROIE and OmniDocBench comparison corresponding to Table 7 . The first metric block evaluates receipt KIE, while the second checks preservation of the original OCR route.
More in this topic
Video-Language Models
Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.
See all 167 papers