Organizations: Keio University, 3-14-1, Kohoku Ward, Yokohama, Kanagawa 223-8522, Japan · National Institute of Informatics Research and Development Center for Large Language Models, 1-1-1, Hitotsubashi, Chiyoda, Tokyo 100-0003, Japan · National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo 101-8430, Japan
In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core attention mechanisms of existing Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences. ABMamba extends Deep State Space Models as its language backbone, replacing the costly quadratic attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes videos across multiple temporal resolutions. On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.
Figures & tables
Figure 1: A typical use case of ABMamba. ABMamba generates relevant and descriptive captions for a given video accompanied by a natural language prompt. Green text highlights the correctly captured contextual information.
Figure 2: The architecture of ABMamba. Given a video with a language prompt, the model generates a caption that concisely describes the visual content. The model consists of a vision encoder, the aligned hierarchical bidirectional scan module, and a Mamba based LLM.
Figure 3: The overview of Aligned Hierarchical Bidirectional Scan (AHBS) module. (a) The module consists of a dimension-wise token compression, projector, AHBS, and temporal token compression. (b) The AHBS explicitly models intricate temporal dynamics through multi-resolution parallel bidirectional scan.
Dataset
Method
Size
BLEU1 ↑
BLEU4 ↑
ROUGE ↑
CIDEr ↑
METEOR ↑
PAC-S ↑
Throughput ↑
(tokens/s)
VATEX
Proprietary
Gemini - 1.5 - Pro
–
22.8
4.5
19.3
14.2
8.9
39.2
–
Fully open MLLMs
Video - ChatGPT
7B
14.8
2.4
25.0
9.9
10.6
39.4
35.2
Video - LLaVA
7B
67.3
24.5
44.9
37.7
19.6
40.8
28.4
Table 1: Quantitative comparison between ABMamba and baseline methods on the test sets of the VATEX and MSR-VTT benchmarks. The best score for each metric is shown in bold . We compared our fully open MLLM ( < 4B parameters) against other fully open MLLMs ( < 7B) and small ( < 4B) but not fully open MLLMs.
(i)
(ii)
(a)
xvision
(b)
y
A man cleans a window with a squeegee while outside.
A group of men are throwing axes at tree stumps several feet away.
(c)
ABMamba
A man is cleaning a window with a squeegee.
A group of people are standing in a field and throwing axes at a target.
(d)
InternVL2.5
The man sprays the window with a spray bottle.
The man wearing a blue shirt is aiming at the board.
(e)
LLaVA-OneVision
A man cleans a large glass window, focusing on the lower section.
A man gives a thumbs-up sign while standing on a gravel path, surrounded by people in various attire.
Table 2: Qualitative results of ABMamba and baseline methods on the VATEX benchmark. Rows (a), (b), (c), (d), and (e) show xvision , y , and the captions generated by ABMamba, InternVL2.5, and LLaVA-OneVision, respectively. Note that y denotes one of the reference captions.
Table 6
Setting
Value
Batch size
128
Optimizer
AdamW
LR schedule
Cosine decay
Learning rate
2×10−5
Epoch
2
Warmup ratio
0.03
Table 5: Experimental settings of ABMamba.
Method
Size
VideoMME (w/o sub) ↑
Video-ChatGPT
7B
28.0
Video-LLaVA
7B
30.6
LLaVA-OneVision
7B
40.1
InternVL2.5
2.2B
27.6
VideoLLaMA3
2B
27.2
ABMamba (Ours)
3.6B
29.4
Table 6: Comparison of VideoMME (without subtitle) scores across various models.
Method
Size
Initial Mem.(MB)
Peak Mem. (MB)
Mem. Increase (MB)
Throughput (tokens/s)
Video-ChatGPT
7B
13,440
15,505
2,066
38.1
Video-LLaVA
7B
28,104
30,652
2,548
28.9
LLaVA-OneVision
7B
15,813
21,004
5,191
24.8
InternVL2.5
2.2B
4,593
12,698
8,105
31.5
Video-LLaMA3
2B
3,751
26,705
22,955
33.5
ABMamba (Ours)
3.6B
7,088
7,570
482
95.4
Table 7: Comparison of inference-time memory usage and throughput on MSR-VTT. ABMamba achieves both significantly reduced memory overhead and improved throughput.
(a)
xvision
(b)
y
A young man is talking on a microphone to the camera, before and after a scene of kids jay-walking in front of cars is shown.
(c)
ABMamba
A man in a red shirt talks to a woman in a blue shirt in a hallway.
(d)
InternVL2.5
The man is talking to the camera.
(e)
LLaVA-OneVision
A person is seen holding a microphone in a school hallway, with other students walking by.
Table 8: Failure case from ABMamba and baselines on the VATEX benchmark. Panel (a) shows xvision , while panels (b), (c), and (d) present the captions generated by ABMamba, InternVL2.5, and LLaVA-OneVision, respectively. Note that y denotes the reference captions.
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1