cs.CVOct 24, 2026

MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding

Authors: Quang-Trung Truong, Thanh-Nghia Nguyen, Phu-Phong Vo, Jaeyeon Kim, Cheng Chen, Jie Zhang, Binh-Son Hua, Duc Thanh Nguyen, 

Organizations: Hong Kong University of Science and Technology, Hong Kong SAR · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science Technology and Research (A*STAR), Singapore, Singapore · University of Information Technology, Vietnam National University, Ho Chi Minh city, Vietnam · Trinity College Dublin, Ireland · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Marine video understanding is crucial for a wide range of real-world applications. However, existing marine video datasets suffer from limitations, including the limited number of data samples, fine-grained text descriptions, and detailed visual attributes. To address these limitations, we introduce a large-scale, video-text dataset featuring rich, in-the-wild visual attributes tailored for complex marine environments. Our dataset comprises 2,000 video-text pairs annotated with 13 attributes covering up to four objects per scene, object visual/geometric properties, behaviors, environmental context, and camera motion, spanning diverse lighting conditions, times of day, and 7 countries. From these annotations, we generate 16,080 video-question pairs (12,080 multiple-choice, 4,000 open-ended). Furthermore, we benchmark both proprietary and open-source video-LLM models on video question answering and video captioning tasks. We find that (i) proprietary models often achieves superior performances; (ii) VLMs handle object motion well but struggle with complex camera motion; (iii) closed-source models struggle to perceive camera motion; and (iv) VLMs struggle to reason in static cameras.

Figures & tables

Figure 1: Annotation Highlights. We build a dataset of 2,000 video–text pairs, containing fine-grained motion dynamics, including diverse object and camera poses, captured across 7 countries. Top : We show video–attribute pairs, highlighting object motion and camera motion to reveal the motion dynamics in marine scenarios. Bottom : We benchmark our dataset on video understanding tasks, ranging from captioning to question-answering. We show how Motion Dynamics , Multiple Objects and Others contribute to different dimensions.
DatasetTimeDomain#SentenceCaption#Video-textCap SourceAttributesPrompt
 (hours)  LengthPairs ObjectMotionCameraComplexity
WebVid [ 2 ]13Kopen-122.5Mgenerated✗✗✗low
InternVid-VTT [ 26 ]76Kopen110-207.1Mgenerated✗✗✗medium
MiraData [ 7 ]16K----generated✗✗✗low
MovieBench [ 29 ]69.2movie---generated✓✓✓medium
MSR-VTT [ 31 ]41.2open19.3200Kmanual✗✗✗low
Table 1: Comparison with existing mainstream video caption datasets. MarineLife-16K features extensive manual annotations for underwater video data, incorporating several attributes, i.e., object awareness, object motion, and camera motion in a hierarchical manner. Note, MovieRecapsQA [ 19 ] and MOMA-QA [ 4 ] datasets are designed for QA tasks, but our dataset is larger than the QA counterparts.
Figure 2: Illustration of motion hallucination. The red texts represent hallucinated camera motion by Gemini 3.1 [ 6 ] , demonstrating failure to capture camera movement. We capture camera motion ground truths with ❶ - ❷ : wide-angle , ❸ - ❹ : close-up , ❺ : bird’s-eye perspective .
Figure 3: A caption engine. We introduce a three-stage, human-in-the-loop caption engine that utilizes a multi-agent framework to generate high-quality video captions. ❶ Fine-grained attributes are manually annotated by human experts. ❷ A LMM captioner agent inputs video sequences to generate initial synthetic captions, which a reviewer agent subsequently scores against the human-verified attributes ❸ A refiner agent filters the outputs, correcting low-confidence synthetic captions to ensure factual alignment.
Figure 4: An example of annotated attributes.
Video ChallengeVisual Attribute
Object-Centric
Heterogeneous Object. Object regions have distinct colors.Color
Shape Complexity. The object has complex boundaries, such as thin parts and holes.Size/Shapes
Appearance Change. Noticeable appearance variation, due to illumination changes and relative camera-object rotation.Lighting , Camera Motion , Size/Shape
Low Resolution. The ratio between the average object bounding-box area and the image area is smaller than 0.1.Size/Shape
Object & Camera Pose
Table 2: Taxonomy of video challenges and link to visual attributes. We extend attributes [ 16 ] to video-text understanding.
Figure 5: Statistics on the frequency of text-caption of MarineLife-16K. We show frequencies by instance object types (from object 1 to 4) (Top) , video quality (Top Right) , the size/shape, color, and texture of the central object (Middle Left) , camera motion (angle and position) (Bottom Right) , and object motion (Bottom Left) .
Figure 6: Human study. We recruit 6 annotators to score LLM predictions across 100 video inputs, requiring 180 total labor hours. We observe both human and LLM judges are consistent when evaluating open-ended QA, which confirms the effectiveness of using LLMs as judges for QA tasks.
Figure 7: Overview of task dimensions of MarineLife-16K.
ModelBackboneMBC-soMBC-moPMTPDOOBKSM-eSM-lOEMU
Open-source VLMs (7B - 8B) 
Video-Llama3-7B [ 33 ]2D RoPE96.666.676.663.39073.335.32.2
LLaVa-NeXT-Video [ 9 ]CLIP8056.7406083.323.311.82.14
LLaVA-OneVision-7B [ 8 ]SigLIP/Qwen9063.36053.38083.329.42.11
Qwen2.5-VL-7B [ 28 ]2D RoPE/Qwen96.766.766.77096.766.729.42.43
InternVL3-8B [ 25 ]V2PE/Qwen96.783.373.37093.373.352.91.36
Table 3: Model performance on video question answering. This table reports model performance on our benchmark across MBC, PMT, OBK, SM and PDO metrics. Camera motion-specific metric is challenging for VLM, highlighted in red .
ModelStatic CameraMoving Camera
BEHAHECFBEHAHECF
Open-source VLMs (7B - 8B)
Video-Llama3-7B [ 33 ]3.132.12.63.132.12.43
LLaVa-NeXT-Video [ 9 ]2.92.712.42.431.722.42
LLaVA-OneVision-7B [ 8 ]3.12.293.143.11.853.14
Qwen2.5-VL-7B [ 28 ]3.432.712.713.142.582.57
Table 4: Comparison of models across static and moving camera settings. Camera motion-specific metric is challenging for VLM, highlighted in red .
Figure 8: Model performance of VLMs on static and moving camera settings.
More in this topic

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 116 papers

Explore similar work

CardsList

  1. MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning

    Jul 27, 2026Tuan-An To, Yuk-Kwan Wong, Tuan-Anh Vu +2Visual Question AnsweringVideo Understanding

    The Hong Kong University of Science and Technology, Hong Kong, China · University of California, Los Angeles CA, USA · University of Electronic Science and Technology of China, China

  2. MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments

    Apr 9, 2026Xingming Liao, Ning Chen, Muying Shu +5VLM EvaluationVisual Question Answering

    Guangdong University of Technology, Guangzhou, China

  3. MotionAtlas: Detailed Region Captioning for Motion-Centric Videos

    Jun 28, 2026Weisong Liu, Haochen Wang, Kuan Gao +8Video CaptioningVideo-Language Models

    Institute of Automation, Chinese Academy of Sciences · Shanghai Jiao Tong University · Peking University +3