cs.CVFeb 18, 2026
SaveReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
Organizations: Keio University · NII LLMC · NII
Abstract
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this study, we focus on video understanding by MLLMs. This task is challenging because processing a full stream of RGB frames is computationally intractable and highly redundant, as self-attention have quadratic complexity with sequence length. In this paper, we propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations. A sparse set of RGB keyframes is retained for appearance, while temporal dynamics are encoded as a motion representation, removing the need for sequential RGB frames. These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding. To refine the noise and low fidelity of block-based motions, we introduce a module to denoise and generate a fine-grained motion representation. Furthermore, our model compresses these features in a way that scales linearly with sequence length. We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks. ReMoRa outperformed baseline methods on multiple challenging benchmarks, including LongVideoBench, NExT-QA, and MLVU.
Figures & tables
Figure 1: Overview of ReMoRa. Our method utilizes compressed video representations, which naturally separates each video into keyframes and compressed inter-frame redundancies. From these, we extract motions that are lightweight but noisy and coarse. Our model then refines these motions into clean and fine-grained representations that preserve efficiency while approaching the fidelity of dense optical flow.
Figure 2: Architecture of ReMoRa : The model operates directly in the compressed video representation for long-video understanding. (a) It consists of an image encoder, the Refined Motion Representation (RMR) module, the Hierarchical Motion State Space (HMSS) Module, and a pretrained LLM. Each clip is decomposed into group of pictures (GOPs) with a single I-frame and several P/B frames represented by motion vectors. The image encoder (Enc.) extracts patch embeddings from I-frames, while the RMR module converts coarse motion vectors into dense, high-fidelity representations. (b) The HMSS module fuses the refined motions and appearance features within each GOP and models long-range dependencies across GOPs through a state space model, enabling linear-time temporal reasoning before alignment with the LLM.
| Method | LLM backbone | LongVideoBench | NExT-QA | MLVU | VideoMME | Perception Test | Average |
|---|---|---|---|---|---|---|---|
| LLaMA-VID [ 28 ] | Vicuna-7B | – | – | 33.2 | – | 44.6 | 38.9 |
| Video-LLaVA [ 30 ] | Vicuna-7B | – | 62.6 | 47.3 | 40.4 | 44.3 | 48.7 |
| PLLaVA [ 58 ] | Vicuna-7B | 40.2 | 68.2 | 47.3 | 44.3 | – | 50.0 |
| VideoChat2 [ 27 ] | Vicuna-7B | 36.0 | – | 54.6 | 47.9 | 47.3 | 46.5 |
| LLaVA-NeXT-Video [ 69 ] | Qwen1.5-7B | 43.5 | – | – | 46.5 | 48.8 | 46.3 |
| Video-LLaMA2 [ 9 ] | Mistral-7B | – | 51.7 | 48.5 | 47.9 | 51.4 | 49.9 |
Table 1: Quantitative comparison of recent video MLLMs across multiple benchmarks. ReMoRa achieved the best performance on LongVideoBench, NExT-QA, and MLVU, and obtains the highest overall average score of 69.8, while remaining highly competitive on VideoMME and Perception Test. Bold indicates the best performance, and underline indicates the second best in each column.
| Method | MSVD-QA | ActivityNet-QA | ||
|---|---|---|---|---|
| Acc. | Score | Acc. | Score | |
| FrozenBiLM [ 61 ] | 32.2 | – | 24.7 | – |
| VideoLLaMA [ 66 ] | 51.6 | 2.5 | 12.4 | 1.1 |
| LLaMA-Adapter [ 68 ] | 54.9 | 3.1 | 34.2 | 2.7 |
| VideoChat [ 25 ] | 56.3 | 2.8 | 26.5 | 2.2 |
| Video-ChatGPT [ 37 ] | 64.9 | 3.3 | 35.2 | 2.7 |
Table 2: Quantitative comparison on open-ended video question-answering benchmarks. ReMoRa attains the best Accuracy and Score on ActivityNet-QA, surpassing the previous best baseline model, and achieves a highly competitive Score on MSVD-QA with strong Accuracy. Bold indicates the best performance, and underline indicate the second best. Acc. denotes the Accuracy.
Figure 3: Qualitative comparison between ReMoRa and LLaVA-Video on NExT-QA. In both examples, ReMoRa correctly answers questions about fine-grained, temporally contextualized human actions and object motions, while LLaVA-Video fails, highlighting ReMoRa’s superior use of motion cues for fine-grained action understanding.
| # Frames | Sampling | VideoMME | NExT-QA | |
|---|---|---|---|---|
| (a) | 64 | CVR | 64.3 | 84.2 |
| (b) | 32 | CVR | 61.9 | 82.7 |
| (c) | 16 | CVR | 58.5 | 81.6 |
| (d) | 64 | Uniform | 62.4 | 82.8 |
Table 3: Ablation study on sampling and frame selection strategies. Using our Compressed Video Representation (CVR)-aware selection of 64 I-frames (a) achieved the best performance. Reducing the number of I-frames degrades results, especially on VideoMME, indicating the importance of sufficient temporal coverage. Simple uniform sampling (d) leads to lower scores, showing that our method is more effective than uniform sampling for long-video understanding.
| Model | VideoMME | NExT-QA | |
|---|---|---|---|
| (a) | Full | 64.3 | 84.2 |
| (f) | w/o Align | 63.4 | 82.2 |
| (g) | w/o RMR | 62.1 | 82.0 |
Table 4: Ablation study on the Refined Motion Representation (RMR) module. Removing optical-flow–based pretraining (f) or the RMR module (g) consistently degrades performance, showing that the RMR module with optical-flow pretraining is important for robust video understanding when using motion vectors.
| Model | VideoMME | NExT-QA | |
|---|---|---|---|
| (a) | HMSS (Ours) | 64.3 | 84.2 |
| (h) | Cross-attn | 62.5 | 81.9 |
| (i) | Add | 61.3 | 81.5 |
Table 5: Ablation study on the GOP aggregation strategy. Model (a) outperforms variants that rely on simple cross-attention (h) or naive additive fusion (i) on both benchmarks, highlighting the importance of structured temporal modeling for GOP integration.
| Model | Samples/s | Tokens/s | Max memory |
|---|---|---|---|
| LLaVA-Video [ 70 ] | 0.53 | 31.78 | 23.21 |
| BIMBA [ 15 ] | 0.39 | 25.16 | 10.60 |
| ReMoRa (ours) | 0.40 | 24.45 | 10.59 |
Table 6: Throughput and peak GPU memory usage for different video MLLMs. ReMoRa achieves comparable samples per second and tokens per second throughput to BIMBA while matching its memory footprint, and it reduces peak memory usage by more than half compared with LLaVA-Video. Note that max memory is in GB.
| Optimizer | AdamW |
|---|---|
| Learning rate | 2e-5 |
| Batch size | 32 |
Table 7: Hyperparameters used in our experiments.
Figure 4: Further qualitative comparison between ReMoRa and LLaVA-Video on LongVideoBench. In both examples, ReMoRa correctly answers questions that require integrating spatial details with long-range temporal understanding, such as tracking how the scene and objects change over time and consistently identifying the person involved in the activity, while the baseline model fails.
| Error mode | # instances |
|---|---|
| Spatial comprehension error | 22 |
| Temporal comprehension error | 21 |
| Motion comprehension error | 14 |
| Annotation error | 10 |
| Total | 67 |
Table 8: Categorization of failure modes based on 67 annotated error instances from 50 randomly sampled cases where our model failed and LLaVA-Video succeeded on the NExT-QA benchmark. A single case may belong to multiple error modes.
Figure 5: Example of scene-aware video preprocessing. Frames 0 and 18 are scene-adaptive I-frames used as keyframes, and the remaining frames are P/B-frames with overlaid codec motion vectors.