COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models

Chenghua Zhu; Zhaolu Kang; Qifan Shi; Siyan Wu; Kehan Jiang; Lei Wei; Lianyu Hu; Guangyuan Dong

cs.CVAug 21, 2026

COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models

Authors: Chenghua Zhu, Zhaolu Kang, Qifan Shi, Siyan Wu, Kehan Jiang, Lei Wei, Lianyu Hu, Guangyuan Dong,

Organizations: Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School Peking University Shenzhen, China

Abstract

Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile. The core bottleneck is not only sparse frame sampling, but also the lack of a complete temporal modeling pipeline for explicitly representing frame-to-frame change, enabling appearance-motion interaction, and optimizing temporal direction sensitivity. We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation, appearance-motion fusion, and direction-aware optimization. Architecturally, COMET introduces a temporal motion branch built on Taylor frame differences and injects its motion evidence into the appearance stream via temporal attention bias-enhanced cross-attention. For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal and strengthens the model's use of directional motion patterns encoded by the temporal motion branch. The method achieves consistent overall improvements with a pronounced motion-temporal bias: on Qwen3-VL-8B, action-centric tasks (STAR, SSv2) improve by 4.9% on average, temporal reasoning tasks (NExT-QA, CLEVRER, LLaVA-178K) by 2.1% over BL-GRPO, while static perception tasks (PerceptionTest) remain on par. The same gain pattern also transfers to InternVL2.5-8B, indicating that COMET generalizes across model families.

Figures & tables

Figure 1. Frame-independent encoding versus explicit temporal representation. (a) sparse RGB frames leave motion implicit. (b) frame-to-frame change makes temporal evidence explicit. A comparison between frame-independent video encoding and an explicit representation of frame-to-frame temporal changes.

Figure 2. Overview of COMET instantiated on Qwen3-VL-2B. (a) dual-branch architecture with DeepStack fusion. (b) TAB-enhanced gated cross-attention. An overview of the COMET framework, including its dual-branch visual encoder, multi-level fusion, and motion-aware cross-attention module.

Figure 3. Temporal-specialized training pipeline with TPD and TC-GRPO. An overview of the two-stage training pipeline, consisting of temporal prior distillation and temporal contrast GRPO.

Figure 4. Taylor-style motion maps preserve temporal direction: reversing the video flips motion direction while largely preserving appearance. A qualitative comparison of forward and reversed videos showing the directional information captured by Taylor-style motion maps.

Figure 5. Radar plot of Table 1 . Colors denote Action , Reasoning , and Perception . A radar chart comparing model performance across action, reasoning, and perception task categories.

	Method	Stage	STAR	PercepTest	NExT-QA	CLEVRER	LLaVA-178K	SSv2	Action	Reasoning	Perception	Avg
Qwen3-VL-2B
	Pretrained	0	60.7	60.3	78.8	59.9	69.8	55.8	58.3	69.5	60.3	64.2
	BL-SFT	I	66.2	64.1	80.8	64.1	73.3	61.8	64.0	72.7	64.1	68.4
	Ours-TPD	I	70.7	63.8	82.1	66.9	75.4	67.1	68.9	74.8	63.8	71.0
	BL-GRPO	II	69.0	66.4	82.2	67.4	75.1	64.8	66.9	74.9	66.4	70.8
	Flow4Agent (SAMFlow) † ( Liu et al., 2025 )	X	70.9	66.8	82.7	68.7	75.9	66.9	68.9	75.8	66.8	72.0

Table 1. Main results on six benchmarks and subsets. Stage 0: pretrained; I: SFT/TPD; II: GRPO/TC-GRPO; X: external baselines. † Reproduced under our setup. Best in bold , second-best underlined . Action =avg(STAR, SSv2), Reasoning =avg(NExT-QA, CLEVRER, LLaVA-178K), Perception =PercepTest.

Configuration	STAR	PT	NQA	CLV	LLaVA	SSv2	Avg	$\Delta$
Baseline (BL-SFT)	75.5	68.0	85.4	71.0	78.5	68.8	74.5	—
COMET w/o TAB	78.4	67.7	86.1	72.8	79.6	72.1	76.1	+1.6
Full COMET	79.0	67.6	86.3	73.2	79.9	72.8	76.5	+0.4

Table 2. Component ablation of COMET (Qwen3-VL-8B, Stage I).

\Delta

denotes the gain over the previous row.

Figure 6. Fusion gate dynamics during Stage I. (a) Gate opening $\tanh(\alpha_{l})$ by level. (b) Injection ratio relative to ViT-A features. Two plots showing the evolution of fusion gates and motion-injection ratios during Stage I training.

	Components	Qwen3-VL-8B	InternVL2.5-8B
Ours	Taylor ViT-B	+143.3 M (+1.6 %)	+100.7 M (+1.2 %)
Ours	Fusion modules	+37.8 M $\times$ 4 (+1.7 %)	+28.3 M $\times$ 1 (+0.4 %)
Flow	SAMFlow ( Zhou et al., 2024 )	+650 M (+7.4 %)	+650 M (+8.0 %)
Flow	MegaFlow ( Zhang et al., 2026 )	+936 M (+10.7 %)	+936 M (+11.6 %)

Table 3. Parameter overhead on 8B-scale backbones. Ours : Taylor ViT-B core attention parameters and fusion modules. Flow : external optical-flow estimators.

Optimization	STAR	PT	NQA	CLV	LLaVA	SSv2	Avg	$\Delta$
Ours-TPD (Stage I)	79.0	67.6	86.3	73.2	79.9	72.8	76.5	—
Std. GRPO (7+0)	80.6	69.0	87.0	75.5	81.2	74.7	78.0	+1.5
TC-GRPO (6+1)	81.9	69.3	87.6	76.3	81.7	76.2	78.8	+2.3
TC-GRPO (5+2)	81.3	69.1	87.3	75.9	81.5	75.3	78.4	+1.9
TC-GRPO (4+3)	80.5	68.5	86.7	75.0	80.8	74.1	77.6	+1.1

Table 4. Optimization ablation on COMET (Qwen3-VL-8B; group size = 7). All variants start from the same Stage I checkpoint.

\Delta

denotes the gain over Ours-TPD.

Figure 7. Forward vs. reversed accuracy during TC-GRPO training (Qwen3-VL-8B). A line chart showing forward-video and reversed-video accuracy during TC-GRPO training.

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 117 papers

Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting
Aug 12, 2026Xikai Sun, Kebin Liu, Haotian Wang +3Video Reasoning VLM Reasoning
Tsinghua University, Beijing, China · JD Logistics, Beijing, China
Temporal-Aware Reasoning Optimization for Video Temporal Grounding
Jun 8, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding Temporal Reasoning
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
Feb 18, 2026Daichi Yashima, Shuhei Kurita, Yusuke Oda +1Long-Video Understanding Multimodal Large Language Models
Keio University · NII LLMC · NII

Aug 12, 2026J/K move · Enter open · S save

Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

Xikai Sun, Kebin Liu, Haotian Wang, Li Liu+2

Tsinghua University, Beijing, China · JD Logistics, Beijing, China

Video Reasoning VLM Reasoning Visual Prompt Learning

Open paper AskPDF arXiv

Motion-centric video reasoning is fundamental to interactive applications such as robotic manipulation and autonomous navigation. However, multimodal large language models (MLLMs) typically process videos through sparse uniform sampling to control visual-token and attention costs. This strategy may discard critical transitions between sampled frames, limiting reasoning about object movement, collisions, and causal interactions. To mitigate this issue, we propose Motion-as-Prompt (MaP), a track-guided cross-frame visual prompting framework. MaP recovers dense point trajectories, selects motion-informative frames, and marks the trajectories accumulated between consecutive sampled frames directly onto the visual inputs, making otherwise hidden displacement, direction changes, and interactions observable to frozen MLLMs. Experiments on CLEVRER and Something-Something-v2 show that MaP consistently improves average motion-reasoning accuracy, yielding gains of 4.2% and 8.9% for GPT-5.5, respectively. Notably, these improvements are obtained without degrading non-motion understanding, highlighting the robustness of MaP. These results demonstrate that MaP provides a simple and effective solution for enhancing motion-centric video reasoning without model training or architectural modification. Project page:https://github.com/SunVictor23/MaP.

Abstract

Video Understanding

Explore similar work

Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

Temporal-Aware Reasoning Optimization for Video Temporal Grounding

ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding