cs.CVMar 6, 2026

MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies

Authors: Howard H. Qian, Kejia Ren, Yu Xiang, Vicente Ordonez, Kaiyu Hang

Organizations: Department of Computer Science Rice University · Department of Computer Science University of Texas at Dallas

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Rigid bodies constitute the smallest manipulable elements in the real world, and understanding how they physically interact is fundamental to embodied reasoning and robotic manipulation. Thus, accurate detection, segmentation, and tracking of moving rigid bodies is essential for enabling reasoning modules to interpret and act in diverse environments. However, current segmentation models trained on semantic grouping are limited in their ability to provide meaningful interaction-level cues for completing embodied tasks. To address this gap, we introduce MotionBit, a novel concept that, unlike prior formulations, defines the smallest unit in motion-based segmentation through kinematic spatial twist equivalence, independent of semantics. In this paper, we contribute (1) the MotionBit concept and definition, (2) a hand-labeled benchmark, called MoRiBo, for evaluating moving rigid-body segmentation across robotic manipulation and human-in-the-wild videos, and (3) a learning-free graph-based MotionBits segmentation method that outperforms state-of-the-art embodied perception methods by 37.3% in macro-averaged mIoU on the MoRiBo benchmark. Finally, we demonstrate the effectiveness of MotionBits segmentation for downstream embodied reasoning and manipulation tasks, highlighting its importance as a fundamental primitive for understanding physical interactions.

Figures & tables

Figure 1: The top row shows a robot physically interacting with a variety of complex composite objects, each constructed from colored blocks that have been glued together. The bottom row highlights the output of a standard semantic segmentation model, which incorrectly over-segments the objects. In contrast, the proposed MotionBits segmentation correctly groups composite objects. Example real-world robotic applications requiring accurate MotionBits segmentation are shown in Fig. 6 .
Figure 2: Illustration of spatial twist for two rigid objects. Transparent objects represent initial positions, and solid objects represent positions after motion. The translations of body frames are represented by purple linear velocity vectors υ{x}\upsilon_{\{x\}} . Although υ{a1}≠υ{a2}\upsilon_{\{a_{1}\}}\neq\upsilon_{\{a_{2}\}} and υ{b1}≠υ{b2}\upsilon_{\{b_{1}\}}\neq\upsilon_{\{b_{2}\}} , transforming them to the fixed world frame {s}\{s\} yields identical linear velocities υ{s}x\upsilon_{\{s\}}^{x} for frames on the same rigid object xx but distinct linear velocities for different rigid objects. Intuitively, the instantaneous motion observed at the world frame will appear identical for body frames on the same rigid object, regardless of their local motions, but will differ across rigid bodies.
Track#ScenesFrames/SceneMasks/SceneSource DatasetActions and Motions
Robotic Manipulation27015.9±0.715.9\pm 0.75.2±1.55.2\pm 1.5BridgeData V2 [ 35 ]Push, Grasp, Pick-&-Place
Human-in-the-Wild7911.0±0.011.0\pm 0.010.3±8.610.3\pm 8.6SA-V [ 2 ]Human-Object Interactions
Table 1: Statistics of the MoRiBo benchmark for evaluating moving rigid-body segmentation. Each video has resolution of 480 ×\times 640 or 640 ×\times 480, and hand-annotated segmentation masks are provided for the last frame of each video.
Figure 3: Examples from the new MoRiBo benchmark across both tracks, robotic manipulation and human-in-the-wild. Hand-labeled final-frame segmentation masks of moving rigid bodies are provided for every video.
Figure 5
Figure 4: Our learning-free graph-based method for online MotionBits segmentation.
Figure 5: Qualitative comparison of moving rigid-body segmentation on the two-track MoRiBo benchmark between Qwen2.5-VL (QwenVL) [ 25 ] , Segment Any Motion in Videos (SAMIV) [ 8 ] and our method.
 OverlapBoundary
 MethodPRF1\text{F}_{1}mIoUPRF1\text{F}_{1}
Robotic ManipulationVideoLLaMa [ 26 ]47.29.28.67.443.88.47.7
InternVideo [ 27 ]49.517.715.013.944.214.213.8
QwenVL [ 25 ]19.128.724.621.517.725.121.9
SAMIV [ 8 ]69.531.027.024.361.627.225.8
OCLR Flow [ 9 ]52.516.924.410.223.014.317.8
Table 2: Quantitative results on the MoRiBo benchmark (%). Our method surpasses alternatives across both the Robotic Manipulation and Human-in-the-Wild tracks.
Figure 6: Application examples of MotionBits for downstream tasks in a tabletop scene where a robot autonomously interacts with complex, composite objects. QwenVL [ 25 ] fails to produce an accurate response in the VQA task. However, using our MotionBits segmentation as overlaid marks [ 14 ] improves VQA response accuracy. Comparing object segmentation among SAMIV [ 8 ] , SAM [ 6 ] , and our MotionBits method shows that incorporating rigid-body motions as segmentation cues gives more accurate object-level insight. For the tower stacking task, over-segmentation of the composite objects by SAM [ 6 ] generates robot actions that result in grasp and stack failures. In contrast, our method’s MotionBits segmentation enables successful grasping and stacking.
Method# Stacked ObjsSRDescription of Failures
SAM [ 6 ]40 / 10Grasp & Stack Failures
SAMIV [ 8 ]00 / 10No Segmentation
QwenVL [ 25 ]20 / 10Poor/No Segmentation
Ours376 / 10Cluttered Grasp Failures
Table 3: Results of the tower stacking task guided by object segmentation masks.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S7: Iterative soft label propagation visualized ( Section 5.2 and line 8 of Algorithm 1 ). First, a set of seed nodes SS is uniformly selected from the set of graph nodes M\mathcal{M} , and each seed node s∈Ss\in S is assigned a different “colored paint”. Then, by continuously pouring the assigned color of paint onto each seed node and allowing the paints to flow and blend naturally according to the structure of graph GG , we capture spatial twist motion regions through a color-coded soft MotionBits segmentation mask.
Figure S8: Examples of Qwen2.5VL-32B [ 25 ] responses to different prompts. The prompt that empirically yielded the most accurate moving rigid-body segmentation results (highlighted) was used to evaluate VLMs on the MoRiBo benchmark, reflected in Section 6.1 .
Figure S9: Additional example of MotionBits segmentation applied to the downstream task of opening a cabinet drawer. When a person interacts with the drawer, our method accurately segments the relevant MotionBits, including the drawer door and the two components of the drawer slider. This segmentation provides interaction-level cues that enable an embodied system to reason about and manipulate the cabinet.
Figure S10: Additional example of MotionBits segmentation applied to the downstream task of manipulating the water bottle handle. When a person manipulates the water bottle handle, our method accurately segments the handle as a MotionBit, which provides interaction-level cues that enable a robot to manipulate the handle in a similar manner.
Figure S11: Extended qualitative comparisons of moving rigid-body segmentation on the Robotic Manipulation track of the MoRiBo benchmark between VideoLLaMa [ 26 ] , InternVideo [ 27 ] , QwenVL [ 25 ] , SAMIV [ 8 ] , OCLR TTA [ 9 ] and our method. These results demonstrate that existing perception methods lack fundamental motion-level grounding, which is essential for embodied systems to reason about and interact with real-world environments. Our method is able to effectively identify moving rigid bodies within a scene.
Figure S12: Extended qualitative comparisons of moving rigid-body segmentation on the Human-in-the-Wild track of the MoRiBo benchmark between VideoLLaMa [ 26 ] , InternVideo [ 27 ] , QwenVL [ 25 ] , SAMIV [ 8 ] , OCLR TTA [ 9 ] and our method. These results demonstrate that existing perception methods lack fundamental motion-level grounding, which is essential for embodied systems to reason about and interact with real-world environments. Our method is able to effectively identify moving rigid bodies within a scene.
Figure S13: More examples from the Robotic Manipulation track of the proposed MoRiBo benchmark ( Section 4 ).
Figure S14: More examples from the Human-in-the-Wild track of the proposed MoRiBo benchmark ( Section 4 ).

Explore similar work

CardsList
Apr 24, 2026

GenMatter: Perceiving Physical Objects with Generative Matter Models

Human visual perception offers valuable insights for understanding computational principles of motion-based scene interpretation. Humans robustly detect and segment moving entities that constitute independently moveable chunks of matter, whether observing sparse moving dots, textured surfaces, or naturalistic scenes. In contrast, existing computer vision systems lack a unified approach that works across these diverse settings. Inspired by principles of human perception, we propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles (small Gaussians representing local matter), and groups particles into clusters capturing coherently and independently moveable physical entities. We develop a hardware-accelerated inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and groupings. Our model operates on different kinds of inputs (random dots, stylized textures, or naturalistic RGB video), enabling it to work across settings where biological vision succeeds but existing computer vision approaches do not. We validate this unified framework across three domains: on 2D random dot kinematograms, our approach captures human object perception including graded uncertainty across ambiguous conditions; on a Gestalt-inspired dataset of camouflaged rotating objects, our approach recovers correct 3D structure from motion and thereby accurate 2D object segmentation; and on naturalistic RGB videos, our model tracks the moving 3D matter that makes up deforming objects, enabling robust object-level scene understanding. This work thus establishes a general framework for motion-based perception grounded in principles of human vision.

Eric Li, Arijit Dasgupta, Yoni Friedman +5

1MIT CSAIL · 2MIT BCS

Video Object SegmentationObject-Centric Representation LearningStructure from Motion3D Scene Understanding
AskPDF
May 20, 2026

ΔΔynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos

Inferring rigid-body physical states and properties from monocular videos is a fundamental step toward physics-based perception and simulation. Existing approaches assume specific underlying physical systems, object types, and camera poses, making them unable to generalize to complex real-world settings. We introduce ΔΔYNAMICS, a vision-language framework that uses language as a unified representation of rigid-body dynamics. Instead of directly predicting parameters, ΔΔYNAMICS generates scene configurations in a structured text format for physics simulation. We enhance the model's generalization by integrating natural language motion reasoning and leveraging optical flow as a semantic-agnostic input. On the CLEVRER dataset, ΔΔYNAMICS achieves a segmentation IoU of 0.30, a 7x improvement over leading VLMs (InternVL3-8B, Qwen2.5-VL-7B and Claude-4-Sonnet). Additionally, test-time sampling and evolutionary search further boost performance by 27% and 120% in segmentation IoU, respectively. Finally, we demonstrate strong transfer to a new dataset of 235 real-world rigid-body videos, highlighting the potential of language-driven physics inference for bridging perception and simulation.

Chia-Hsiang Kao, Cong Phuoc Huynh, Chien-Yi Wang +5

1Cornell University · 2Amazon

VLM ReasoningPhysical Reasoning in Video
AskPDF
Mar 18, 2026

Towards Motion-aware Referring Image Segmentation

Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL

Chaeyun Kim, Seunghoon Yi, Yejin Kim +2

Seoul National University · AIM Intelligence

Referring Image SegmentationMultimodal Contrastive Learning
AskPDF