Organizations: Hong Kong University of Science and Technology, Hong Kong SAR · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science Technology and Research (A*STAR), Singapore, Singapore · University of Information Technology, Vietnam National University, Ho Chi Minh city, Vietnam · Trinity College Dublin, Ireland · Deakin University, Australia · Nanyang Technological University, Singapore, Singapore
Marine video understanding is crucial for a wide range of real-world applications. However, existing marine video datasets suffer from limitations, including the limited number of data samples, fine-grained text descriptions, and detailed visual attributes. To address these limitations, we introduce a large-scale, video-text dataset featuring rich, in-the-wild visual attributes tailored for complex marine environments. Our dataset comprises 2,000 video-text pairs annotated with 13 attributes covering up to four objects per scene, object visual/geometric properties, behaviors, environmental context, and camera motion, spanning diverse lighting conditions, times of day, and 7 countries. From these annotations, we generate 16,080 video-question pairs (12,080 multiple-choice, 4,000 open-ended). Furthermore, we benchmark both proprietary and open-source video-LLM models on video question answering and video captioning tasks. We find that (i) proprietary models often achieves superior performances; (ii) VLMs handle object motion well but struggle with complex camera motion; (iii) closed-source models struggle to perceive camera motion; and (iv) VLMs struggle to reason in static cameras.
Figures & tables
Figure 1: Annotation Highlights. We build a dataset of 2,000 video–text pairs, containing fine-grained motion dynamics, including diverse object and camera poses, captured across 7 countries. Top : We show video–attribute pairs, highlighting object motion and camera motion to reveal the motion dynamics in marine scenarios. Bottom : We benchmark our dataset on video understanding tasks, ranging from captioning to question-answering. We show how Motion Dynamics , Multiple Objects and Others contribute to different dimensions.
Dataset
Time
Domain
#Sentence
Caption
#Video-text
Cap Source
Attributes
Prompt
(hours)
Length
Pairs
Object
Motion
Camera
Complexity
WebVid [ 2 ]
13K
open
-
12
2.5M
generated
✗
✗
✗
low
InternVid-VTT [ 26 ]
76K
open
1
10-20
7.1M
generated
✗
✗
✗
medium
MiraData [ 7 ]
16K
-
-
-
-
generated
✗
✗
✗
low
MovieBench [ 29 ]
69.2
movie
-
-
-
generated
✓
✓
✓
medium
MSR-VTT [ 31 ]
41.2
open
1
9.3
200K
manual
✗
✗
✗
low
Table 1: Comparison with existing mainstream video caption datasets. MarineLife-16K features extensive manual annotations for underwater video data, incorporating several attributes, i.e., object awareness, object motion, and camera motion in a hierarchical manner. Note, MovieRecapsQA [ 19 ] and MOMA-QA [ 4 ] datasets are designed for QA tasks, but our dataset is larger than the QA counterparts.
Figure 2: Illustration of motion hallucination. The red texts represent hallucinated camera motion by Gemini 3.1 [ 6 ] , demonstrating failure to capture camera movement. We capture camera motion ground truths with ❶ - ❷ : wide-angle , ❸ - ❹ : close-up , ❺ : bird’s-eye perspective .
Figure 3: A caption engine. We introduce a three-stage, human-in-the-loop caption engine that utilizes a multi-agent framework to generate high-quality video captions. ❶ Fine-grained attributes are manually annotated by human experts. ❷ A LMM captioner agent inputs video sequences to generate initial synthetic captions, which a reviewer agent subsequently scores against the human-verified attributes ❸ A refiner agent filters the outputs, correcting low-confidence synthetic captions to ensure factual alignment.
Figure 4: An example of annotated attributes.
Video Challenge
Visual Attribute
Object-Centric
Heterogeneous Object. Object regions have distinct colors.
Color
Shape Complexity. The object has complex boundaries, such as thin parts and holes.
Size/Shapes
Appearance Change. Noticeable appearance variation, due to illumination changes and relative camera-object rotation.
Lighting , Camera Motion , Size/Shape
Low Resolution. The ratio between the average object bounding-box area and the image area is smaller than 0.1.
Size/Shape
Object & Camera Pose
Table 2: Taxonomy of video challenges and link to visual attributes. We extend attributes [ 16 ] to video-text understanding.
Figure 5: Statistics on the frequency of text-caption of MarineLife-16K. We show frequencies by instance object types (from object 1 to 4) (Top) , video quality (Top Right) , the size/shape, color, and texture of the central object (Middle Left) , camera motion (angle and position) (Bottom Right) , and object motion (Bottom Left) .
Figure 6: Human study. We recruit 6 annotators to score LLM predictions across 100 video inputs, requiring 180 total labor hours. We observe both human and LLM judges are consistent when evaluating open-ended QA, which confirms the effectiveness of using LLMs as judges for QA tasks.
Figure 7: Overview of task dimensions of MarineLife-16K.
Model
Backbone
MBC-so
MBC-mo
PMT
PDO
OBK
SM-e
SM-l
OEMU
Open-source VLMs (7B - 8B)
Video-Llama3-7B [ 33 ]
2D RoPE
96.6
66.6
76.6
63.3
90
73.3
35.3
2.2
LLaVa-NeXT-Video [ 9 ]
CLIP
80
56.7
40
60
83.3
23.3
11.8
2.14
LLaVA-OneVision-7B [ 8 ]
SigLIP/Qwen
90
63.3
60
53.3
80
83.3
29.4
2.11
Qwen2.5-VL-7B [ 28 ]
2D RoPE/Qwen
96.7
66.7
66.7
70
96.7
66.7
29.4
2.43
InternVL3-8B [ 25 ]
V2PE/Qwen
96.7
83.3
73.3
70
93.3
73.3
52.9
1.36
Table 3: Model performance on video question answering. This table reports model performance on our benchmark across MBC, PMT, OBK, SM and PDO metrics. Camera motion-specific metric is challenging for VLM, highlighted in red .
Model
Static Camera
Moving Camera
BEH
AH
ECF
BEH
AH
ECF
Open-source VLMs (7B - 8B)
Video-Llama3-7B [ 33 ]
3.13
2.1
2.6
3.13
2.1
2.43
LLaVa-NeXT-Video [ 9 ]
2.9
2.71
2.4
2.43
1.72
2.42
LLaVA-OneVision-7B [ 8 ]
3.1
2.29
3.14
3.1
1.85
3.14
Qwen2.5-VL-7B [ 28 ]
3.43
2.71
2.71
3.14
2.58
2.57
Table 4: Comparison of models across static and moving camera settings. Camera motion-specific metric is challenging for VLM, highlighted in red .
Figure 8: Model performance of VLMs on static and moving camera settings.
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.
Recent Vision-Language Models (VLMs) have achieved remarkable success in visual understanding, driven by the growing availability of high-quality image-text pairs. However, the performance of VLMs often degrades in the video domain due to the essential need for temporal understanding and the scarcity of large-scale annotated video data. In this work, we focus on marine video understanding, which brings further challenges: first, it requires substantial domain expertise; and video VLMs usually struggle with localizing and interpreting critical information from marine videos, as the informative events are typically sparse, unpredictable, and unevenly distributed. To address these challenges, we carefully curate the first event-centric marine video understanding dataset called MarineEVT, which features 20K multi-task, video-level visual question-answering pairs spanning multiple dimensions of marine understanding and analysis. Meanwhile, based on MarineEVT, we decompose marine video understanding as an Event-centric Visual Tool-integrated Reasoning process EVT-R1 for short, where we leverage powerful visual tools to drive the model to localize and interpret critical information aligned with visual questions and human intent. To demonstrate its effectiveness, we compare EVT-R1 against 11 SOTA VLMs in different settings. EVT-R1 outperforms the top open-source and top commercial models by 5.22 and 11.09, respectively. MarineEVT and EVT-R1 lay the foundation for ecological discovery and marine education, fostering the development of VLMs capable of interpreting marine dynamics, reasoning about ecological interactions, and supporting sustainable ocean video understanding and analysis.
Tuan-An To, Yuk-Kwan Wong, Tuan-Anh Vu +2
The Hong Kong University of Science and Technology, Hong Kong, China · University of California, Los Angeles CA, USA · University of Electronic Science and Technology of China, China
Fine-grained visual understanding and high-level reasoning in real-world open-water environments remain under-explored due to the lack of dedicated benchmarks. We introduce MARINER, a comprehensive benchmark built under the novel Entity-Environment-Event (3E) paradigm. MARINER contains 16,629 multi-source maritime images with 63 fine-grained vessel categories, diverse adverse environments, and 5 typical dynamic maritime incidents, covering fine-grained classification, object detection, and visual question answering tasks. We conduct extensive evaluations on mainstream Multimodal Large language models (MLLMs) and establish baselines, revealing that even advanced models struggle with fine-grained discrimination and causal reasoning in complex marine scenes. As a dedicated maritime benchmark, MARINER fills the gap of realistic and cognitive-level evaluation for maritime multimodal understanding, and promotes future research on robust vision-language models for open-water applications. Appendix and supplementary materials are available at https://lxixim.github.io/MARINER.
Xingming Liao, Ning Chen, Muying Shu +5
Guangdong University of Technology, Guangzhou, China
We propose MotionAtlas, a system for detailed captioning of motion-centric videos, comprising (1) a dedicated human-annotated benchmark, (2) a scalable, high-quality pipeline to construct training samples, and (3) a family of powerful Video-MLLMs. Unlike conventional global motion captioning datasets, we focus on region-aware motion captioning: given a video and a spatiotemporal mask, the model generates precise descriptions of motion within the target region, thereby alleviating visual clutter and motion entanglement and enabling reliable, quantifiable evaluation. Concretely, we first build MotionAtlas-Bench, a comprehensive benchmark comprising 2,073 multiple-choice questions, meticulously annotated for a curated set of high-quality, motion-centric videos, to evaluate fine-grained motion understanding of the objects in question. Second, we design a rigorous and scalable data pipeline that leverages self-bootstrap refinement to suppress fine-grained hallucinations, yielding 159k high-quality motion captioning data. Third, we design a tailored training data composition strategy, which achieves consistent and substantial performance gains across diverse baseline Video-MLLMs, including Molmo2 and Qwen3-VL. For instance, MotionAtlas-4B surpasses Qwen3-VL-4B by an average of 5.2 percentage points across general motion benchmarks. The benchmark, dataset, and code have been released.
Weisong Liu, Haochen Wang, Kuan Gao +8
Institute of Automation, Chinese Academy of Sciences · Shanghai Jiao Tong University · Peking University +3