PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

Mennatullah Siam

cs.CVSep 2, 2025

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

Authors: Mennatullah Siam

Abstract

Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the referring expression without any temporal reasoning. To address this, we introduce novel motion-centric probing, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the ordering of motion. Consequently, we introduce MoCentric-Bench, a motion-centric benchmark designed to evaluate video MLLMs based on their ability to capture the interaction between motion and language, rather than relying primarily on static cues. We further establish strong single-image baselines that either match or surpass prior methods. Finally, we explore a simple motion-centric adaptation that provides state-of-the-art performance on our MoCentric-Bench. Code is available at https://msiam.github.io/PixFoundationSeries/.

Figures & tables

Figure 1: Left: (Top) Shortcomings of the current referring segmentation benchmarks, where the expression can be captured from a single frame. (Bottom) Illustration of the multi-video layout used in our benchmark as a probe to quantify failures of video MLLMs in capturing motion. Right: Our MoCentric-Bench evaluates the use of motion in visual grounding within videos. Our probing is designed for visual grounding using a static single frame or the reverse of the video. The correct and deceiving ground-truth segmentations are highlighted in red and blue, respectively.

Method	Year	Probing w/ input	Task
RESOUND Li et al. (2018)	ECCV 2018	Object, Scene, People Rep.	Action Recog.
Scene Bias Choi et al. (2019)	CVPR 2019	Mask Out	Action Recog., Temp. Loc., Action Det.
Dyn.Stat. Kowal et al. (2022)	CVPR 2022	Shuffle, Perturb OF	Action Recog., Video Seg. w/o Language
ATP Buch et al. (2022)	CVPR 2022	Single Frame Rep.	Video QA, Video Ret.
Apollo Zohar et al. (2025)	CVPR 2025	Single Frame, Text Only	Video QA
TV-Bench Cores et al. (2024)	Arxiv 2024	Single Frame, Shuffle, Reverse, Text Only	Video QA

Table 1: Video understanding studies , with their respective probing methods. Our work is the first to study visual grounding in videos. Probing w/ input indicates a controlled change in the input to the model towards assessing certain ability or bias. Rep.: representations, OF: Optical Flow, CM: caption matching, DC: direction classification.

Panel of Figure 2 — Figure 2: Qualitative analysis of our proposed automatic keyframe selection from five examples that show a single frame can be sufficient to convey the motion expression without any motion involved. It is mainly conveyed through the use of static cues such as the heading, object type, or position. Expressions of each example are as follows: (a) “jump to the left then jump back”, (b) “dog playing with monkey”, (c) “puppy that overwhelms another puppy”, (d) “cow shaking head and looking at us.” (e) “The little cat walking from behind to the front”. Groundtruth segmentation highlighted in red.

Figure 3: MoCentric-Bench. The mechanism for creating our motion-centric benchmark. Top: Motion existence probe, $\mathcal{P}_{1}$ . We rely on an automatic keyframe selection, using Qwen2.5-VL, to identify a static frame to convey the motion referring expression (i.e., keyframe) and use it to deceive video-MLLMs as fake motion. The output modified video is the repeated keyframe and can be paired with the original video in various layouts to deceive their grounding mechanisms. We show that state-of-the-art models can easily be deceived with these static keyframes. Bottom: Motion ordering probe, $\mathcal{P}_{2}$ . We use GPT-4o to perform an automatic motion referring expression reversal to match the respective reverse of the video. The output modified video and expression can similarly be paired with the original video to deceive video-MLLMs’ grounding mechanisms in their ability to differentiate the original (forward) from the reverse of the video. $[.,.]$ is the horizontal concatenation operator. Ground-truth segmentation of the fake and true referred objects are highlighted in green.

Panel of Table 2 — Table 2: Comparison of our mocentric dataset with visual grounding benchmarks. It confirms that our benchmark maintains close or on-par statistics while introducing the advantage of evaluating MLLMs’ ability to capture local motion. Note that locality here refers to the temporal dimension, where a single frame or non-consecutive keyframes are insufficient to convey the motion expression without a genuine understanding of the underlying motion between them.

Method	Venue	val u			val u & Single frame			Reverse			val u & Reverse
		$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
GPT-5	-	30.0	36.9	33.5	17.8	27.8	22.8	33.9	39.8	36.8	17.5	26.3	21.9
LMPM	ICCV 2023	34.2	40.2	37.2	18.2	26.9	22.6	31.8	38.1	35.0	18.0	25.4	21.7
VidGLAMM	CVPR 2025	43.6	52.7	48.2	18.1	25.1	21.6	49.4	58.4	53.9	28.7	39.3	34.0
RGA	ICCV 2025	53.2	59.1	56.1	23.9	32.5	28.2	57.3	63.3	60.3	32.0	42.1	37.0
Sa2VA	TPAMI 2026	-	-	58.9	23.5	33.5	28.5	55.9	66.4	61.1	31.0	42.9	37.0

Table 3: MoCentric-Bench results based on the synthesized multi-video layout variants of MeVIS dataset val u subset, following probe

\mathcal{P}_{1}

and

l_{2}

layout (i.e., val u & Single frame ), following probe

\mathcal{P}_{2}

and

l_{1}

(i.e., Reverse ) and following probe

\mathcal{P}_{2}

and

l_{2}

(i.e., val u & Reverse ).

\dagger

: indicates the use of the automatic keyframe selection.

\star

: the motion-centric adaptation. It clearly shows a drop in performance for all the models, including our baselines from val u to val u & Single frame , similarly between the Reverse and val u & Reverse . It shows that the state-of-the-art methods and our single-image baselines can not differentiate true from fake motion and do not necessarily understand the ordering of motion. Best results are bolded.

Method	Venue	Backbone/Base MLLM	RefDAVIS-17			MeVIS
			$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
GPT-5	-	-	-	-	25.2	-	-	23.4
ReferFormer	CVPR 2022	VideoSwin-B	58.1	64.1	61.1	29.8	32.2	31.0
LMPM	ICCV 2023	Swin-T	-	-	-	34.2	40.2	37.2
LISA	CVPR 2024	LLaVA-7B	62.2	67.3	64.8	35.1	39.4	37.2
TrackGPT	Arxiv 2023	LLaVA-7B	63.2	59.4	67.0	37.6	42.6	40.1

Table 4: Comparison of recent video MLLMs to our single-image baselines, our motion-centric Sa2VA

\star

and Molmo 2.0

\star

, on RefDAVIS’17 and MeVIS val subset. Our baselines are mostly biased to the static information, yet surpass previously published works.

\dagger

: using automatic keyframe selection.

\star

: the motion-centric adaptation. Best and second-best are bolded and underlined, resp.

Panel of Figure 4 — Figure 4: Qualitative ablation on MoCentric-Bench comparing our strongest baseline, Baseline $\dagger$ , our adapted motion-centric model, Sa2VA $\star$ and concurrent work from VideoGLAMM. Top rows: val u & Reverse , expression is “bear landing from a jump backward over a barricade and walking backward”. Bottom rows: val u & Single frame , expression is “the elephant walking to the left”.

Dataset Source	Variant	Videos	Probe & Layout	Video-Expression pairs
MeVIS (train)	MoCentric Train	3,324	$\mathcal{P}_{1}$ , ( $l_{2},l_{3}$ )	46,102
MeVIS (val u )	val u & Single frame	150	$\mathcal{P}_{1}$ , ( $l_{2},l_{3},l_{4}$ )	2,379
MeVIS (val u )	val u & Blank image	100	$\mathcal{P}_{3}$ , ( $l_{2},l_{3}$ )	1,586
MeVIS (val u )	Reverse	28	$\mathcal{P}_{2}$ , ( $l_{1}$ )	152
MeVIS (val u )	val u & Reverse	56	$\mathcal{P}_{2}$ , ( $l_{2},l_{3}$ )	304
Molmo2Track (Animal)	Animal & Single frame	188	$\mathcal{P}_{1}$ , ( $l_{2}$ )	332

Table 6: Detailed statistics of our synthetically generated dataset.

Exp. #	Forward Motion Expression	Reverse Motion Expression
1	turn and walk away from us	turn and walk towards us
2	the elephant walking to the left	the elephant walking to the right
3	the moving pair of elephants.	the pair of reversing elephants
4	the individual providing food to the lizard	the individual taking food away from the lizard
5	the elephant pressing its nose against the other elephant’s back	the elephant moving its nose away from the other elephant’s back
6	the aircraft moving towards us	the aircrat moving away from us

Table 7: Our synthetic benchmark val u & Reverse . Examples of the original motion expression for the forward motion and its reverse motion expression for the reverse video.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table 12

Method	val u	val u & Single frame		val u & Blank image
	$\mathcal{F}\&\mathcal{J}$	$\mathcal{F}\&\mathcal{J}$	$\Delta$	$\mathcal{F}\&\mathcal{J}$	$\Delta$
LMPM	37.2	22.6	14.6	36.4	0.8
VideoGLAMM	48.2	21.6	26.6	47.9	0.3
RGA	56.1	28.2	27.9	54.7	1.4
Sa2VA	58.9	28.5	30.4	54.8	4.1

Appendix

Table 10: Studying spatial biases. Ablation on the impact of our motion probe,

\mathcal{P}_{1}

, in val u & Single frame variant vs. the spatial bias probe,

\mathcal{P}_{3}

, in val u & Blank image variant.

\Delta

: the difference between the original performance on MeVIS val u and the respective probe performance. It clearly shows that spatial bias has minimal impact relative to the impact of our motion probe, where models are degraded by almost half in the case of val u & Single frame . This is due to the models being misled by the fake motion from the static frame. As such, it confirms the limitations in the current benchmarks and video MLLMs for visual grounding in capturing local motion.

Method	val u	val u & Single frame			Reverse	val u & Reverse
	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Sa2VA	58.9	25.5	38.2	31.8	61.1	26.0	38.7	32.4
Sa2VA $\star$	60.5	26.6	39.1	32.9	65.1	28.4	42.3	35.3

Appendix

Table 11: Ablation of Motion Centric adaptation when averaging the predictions of the two layouts

l_{2}

and

l_{3}

on our MeVIS val u variants.

Panel of Figure 6 — Figure 6: Failure cases of our motion-centric adaptation on the MeVIS variant val u & Reverse , the corresponding referred expression is “the horse that starts with its head facing left and then turns to the right”. The frames shown correspond to frames 45, 55 and 60 near the end of the original video. Top: Groundtruth for the correct region highlighted in red and the deceiving region highlighted in blue. Bottom: The predictions of our motion centric adaptation of Sa2VA are highlighted in red.

Table 16

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Aug 12, 2026

Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

Motion-centric video reasoning is fundamental to interactive applications such as robotic manipulation and autonomous navigation. However, multimodal large language models (MLLMs) typically process videos through sparse uniform sampling to control visual-token and attention costs. This strategy may discard critical transitions between sampled frames, limiting reasoning about object movement, collisions, and causal interactions. To mitigate this issue, we propose Motion-as-Prompt (MaP), a track-guided cross-frame visual prompting framework. MaP recovers dense point trajectories, selects motion-informative frames, and marks the trajectories accumulated between consecutive sampled frames directly onto the visual inputs, making otherwise hidden displacement, direction changes, and interactions observable to frozen MLLMs. Experiments on CLEVRER and Something-Something-v2 show that MaP consistently improves average motion-reasoning accuracy, yielding gains of 4.2% and 8.9% for GPT-5.5, respectively. Notably, these improvements are obtained without degrading non-motion understanding, highlighting the robustness of MaP. These results demonstrate that MaP provides a simple and effective solution for enhancing motion-centric video reasoning without model training or architectural modification. Project page:https://github.com/SunVictor23/MaP.

Xikai Sun, Kebin Liu, Haotian Wang +3

Tsinghua University, Beijing, China · JD Logistics, Beijing, China

Video Reasoning VLM Reasoning Visual Prompt Learning

Ask PDF

Mar 12, 2026

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW

Mohamad Alansari, Naufal Suryanto, Divya Velayudhan +3

Khalifa University, Abu Dhabi, UAE

Multimodal Large Language Models Vision-Language Grounding Referring Video Object Segmentation Temporal Video Grounding

Ask PDF

May 8, 2026

SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding

Multimodal Large Language Models (MLLMs) have made rapid progress in single-video understanding, yet their ability to reason across multiple independent video streams remains poorly understood. Existing multi-video benchmarks rely largely on human-annotated real-world footage, limiting the precision of spatial, temporal, and physical ground truth and making it difficult to diagnose model failures. We introduce SYNCR, a controlled synthetic benchmark for cross-video reasoning with programmatically verified grounding. Built using Habitat, Kubric, and CLEVRER simulator engines, SYNCR contains 4,000 multi-video question-answer pairs grounded in 4,827 unique videos. It evaluates MLLMs across eight tasks spanning four diagnostic pillars: Temporal Alignment, Spatial Tracking, Comparative Reasoning, and Holistic Synthesis. Our zero-shot evaluation of leading open- and closed-weight MLLMs reveals a substantial gap between current models and humans: the best model achieves only 64.5% average accuracy, compared to an 89.5% human baseline. Models perform relatively well on temporal ordering but struggle with precise physical and spatial reasoning, with the best model reaching only 29.8% accuracy on Kinematic Comparison. We further find that parameter scaling and reasoning-specialized post-training improve temporal alignment capabilities, but do not reliably address fine-grained physical tracking or global spatial synthesis. Finally, a sim-to-real correlation analysis suggests that SYNCR tracks model-level trends on a real-world multi-video benchmark.

Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy +1

New York University

Multimodal Grounding Video Reasoning Multimodal Model Evaluation Physical Reasoning in Video Multimodal Understanding

Ask PDF

Abstract

Vision-Language Grounding

Explore similar work

Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding