MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment

Duc Duy Nguyen; Tat-Jun Chin; Minh Hoai

cs.CVFeb 22, 2026

MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment

Authors: Duc Duy Nguyen, Tat-Jun Chin, Minh Hoai

Organizations: Australian Institute for Machine Learning, Adelaide University, Adelaide, SA 5000, Australia

Abstract

We aim to learn a joint representation between inertial measurement unit (IMU) signals and 2D pose sequences extracted from video, enabling accurate cross-modal retrieval, temporal synchronization, subject and body-part localization, and action recognition. To this end, we introduce MoBind, a hierarchical contrastive learning framework designed to address three challenges: (1) filtering out irrelevant visual background, (2) modeling structured multi-sensor IMU configurations, and (3) achieving fine-grained, sub-second temporal alignment. To isolate motion-relevant cues, MoBind aligns IMU signals with skeletal motion sequences rather than raw pixels. We further decompose full-body motion into local body-part trajectories, pairing each with its corresponding IMU to enable semantically grounded multi-sensor alignment. To capture detailed temporal correspondence, MoBind employs a hierarchical contrastive strategy that first aligns token-level temporal segments, then fuses local (body-part) alignment with global (body-wide) motion aggregation. Evaluated on mRi, TotalCapture, and EgoHumans, MoBind consistently outperforms strong baselines across all four tasks, demonstrating robust fine-grained temporal alignment while preserving coarse semantic consistency across modalities. Code is available at https://github.com/bbvisual/ MoBind.

Figures & tables

Figure 1: Proposed framework for motion binding between IMUs and 2D pose sequence from video. Contrastive learning is applied at both the local space, aligning each IMU with its corresponding body-part, and the global space, aligning full-body representations. This representation supports several downstream tasks, including cross-modal retrieval, temporal synchronization, subject and body parts localization, and human action recognition.

Figure 2: Overview of the proposed MoBind. The framework first encodes each IMU stream together with the motion of its corresponding body part, yielding token-level and local-level representations per sensor. These local representations are then aggregated across sensors to form global-level embeddings. The contrastive objective applies at all three levels. In addition, a Masked Token Prediction (MTP) module is used only during training to preserve coarse semantic structure, preventing the model from over-focusing on fine-grained alignment.

	mRi						TotalCapture						EgoHumans
Method	IMU $\rightarrow$ Video			Video $\rightarrow$ IMU			IMU $\rightarrow$ Video			Video $\rightarrow$ IMU			IMU $\rightarrow$ Video			Video $\rightarrow$ IMU
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
IMU2CLIP	0.67	0.88	0.92	0.38	0.69	0.81	0.06	0.20	0.33	0.07	0.17	0.27	0.29	0.51	0.59	0.29	0.51	0.60
DeSPITE	0.57	0.85	0.91	0.32	0.70	0.82	0.03	0.15	0.24	0.03	0.11	0.25	0.54	0.73	0.80	0.54	0.74	0.80
SyncNet	0.77	0.94	0.97	0.75	0.92	0.95	0.51	0.87	0.94	0.54	0.86	0.93	0.74	0.80	0.93	0.71	0.89	0.93
MoBind	0.94	0.99	1.00	0.92	0.99	0.99	0.87	0.96	0.98	0.68	0.91	0.97	0.83	0.93	0.96	0.83	0.93	0.95

Table 1: Cross-modal retrieval performance on the mRi, TotalCapture and EgoHumans datasets. We compare our method against prior contrastive learning baselines in both retrieval directions: IMU

\rightarrow

Video and Video

\rightarrow

IMU. Our method consistently outperforms all others across all ranks, demonstrating strong alignment between modalities. This superior performance is particularly critical for downstream tasks that rely on accurate similarity scores between embedded features.

Panel of Figure 3 — Figure 3: IMU $\rightarrow$ Video retrieval results on mRi (left) and EgoHumans (right). Each example shows the query IMU signal, its corresponding ground-truth video segment, and the top three retrieved video segments. Our method successfully retrieves the ground-truth segment, and the other top-ranked results are also visually similar to the ground truth, demonstrating robust cross-modal alignment.

Method	mRi		TotalCapture		EgoHumans
Method	MAE $\downarrow$	Acc $\uparrow$	MAE $\downarrow$	Acc $\uparrow$	MAE $\downarrow$	Acc $\uparrow$
Random guess	10.72	0.00	10.06	0.00	10.18	0.01
SyncWISE	3.31	0.04	4.07	0.02	3.68	0.02
IMU2CLIP	1.17	0.70	2.32	0.13	3.13	0.44
DeSPITE	1.42	0.63	2.38	0.16	2.58	0.58
SyncNet	0.89	0.76	1.85	0.21	2.93	0.39

Table 2: Synchronization results on three datasets. All models are evaluated on 20-second videos with random temporal offsets sampled uniformly from

[-7,7]

seconds. Top-

k

retrieval is performed with

k=5

. Our method significantly outperforms baselines in both mean absolute error (MAE, in seconds) and accuracy (Acc, within a 200ms tolerance) across all datasets.

Figure 4: Per-action synchronization accuracy on EgoHumans (left) and mRi (right). MoBind achieves sub-50ms error on all EgoHumans actions and under 1s on all mRi actions, despite the challenges posed by repetitive movements and near-duplicate segments. Results confirm MoBind’s robustness across diverse motion types and environments.

Panel of Figure 5 — Figure 5: Examples of body-part localization on EgoHumans. Each column shows the query IMU (top) and the predicted body part with the highest similarity score (bottom), demonstrating accurate identification of sensor placement.

Method	IMU2CLIP	DeSPITE	SyncNet	VIPL	MoBind
Acc	0.9446	0.9759	0.9749	0.9014	0.9812
F1-score	0.9430	0.9743	0.9748	0.8933	0.9801

Table 3: IMU-to-person identification in multi-person scenes from EgoHumans. This experiment evaluates who wears the IMU sensor, and MoBind achieves the highest accuracy and F1 score.

Panel of Figure 6 — Figure 6: Results for the challenging combined task of temporal synchronization and spatial localization. The query $20\mathrm{s}$ IMU signal from the left wrist precedes the video by $6.8\,\text{s}$ . MoBind accurately recovers this offset using a weighted histogram and simultaneously localizes the signal to the correct body part, demonstrating robustness to dynamic and highly repetitive motion.

Method	mRi		TotalCapture
Method	Finetune	1-NN	Finetune	1-NN
UniMTS	0.95	0.36	0.66	0.53
ImageBIND	0.95	0.40	0.72	0.47
EVI-MAE	0.54	0.43	0.63	0.66
Primus	0.82	0.80	0.57	0.67
IMU2CLIP	0.79	0.85	0.60	0.68

Table 4: Human activity recognition results under finetuning and 1-NN setting on the mRi and TotalCapture. For consistency, fine-tuning results are reported at the 5 th epoch.

Combinations			Retrieval		Sync.		Locali zation
global	local	token	I $\rightarrow$ V	V $\rightarrow$ I	MAE	Acc	Locali zation
✓			0.34	0.31	0.74	0.81	0.22
✓	✓		0.77	0.78	0.49	0.86	0.75
✓	✓	✓	0.94	0.92	0.47	0.88	0.81

Table 5: Ablation studies on contrastive objectives on mRi. Results show consistent gains across all tasks as each contrastive level is added, highlighting the effectiveness of the hierarchical design.

Figure 7: Robustness to Sensor Failure. Retrieval performance (R@1 and R@5) under different sensor availability conditions. R@k measures the percentage of queries for which the ground-truth video appears within the top-

k

retrieved results, given the representation computed from a subset of IMU sensors. In general, using more IMUs provides a more complete motion representation and thus improves retrieval accuracy. MoBind remains highly effective even when some sensors are unavailable, demonstrating strong performance under partial sensor input and highlighting its robustness for real-world deployment.

	mRi		TotalCapture
	Finetune	1-NN	Finetune	1-NN
MoBind w/o MTP	0.97	0.76	0.55	0.53
MoBind	0.98	0.86	0.72	0.71

Table 6: Effect of Masked Token Prediction on model performance. The MTP significantly improves action recognition on both datasets, demonstrating its importance for retaining action-level semantics.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 210 papers

AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild
May 21, 2026Baiyu Chen, Zechen Li, Wilson Wongso +5Human Activity Recognition Wearable Sensing
The University of New South Wales · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation
Jul 26, 2026Jianhan Lin, Yuchu Qin, Jiateng Yuan +2Visual-Inertial Odometry Multimodal Sensor Fusion
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China · International Research Center of Big Data for Sustainable Development Goals, Beijing 100094, China · University of Chinese Academy of Sciences, School of Electronic, Electrical and Communication Engineering, Beijing 100049, China +1
Modular Retrieval-Augmented Generalization for Human Action Recognition
Apr 28, 2026Peng Liao, Shangsong Liang, Lin Chen +1Human Activity Recognition
School of Computer Science and Engineering, Sun Yat-sen University · Engineering Research Centre of Applied Technology on Machine Translation and Artificial Intelligence, Macao Polytechnic University

May 21, 2026J/K move · Enter open · S save

AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild

Baiyu Chen, Zechen Li, Wilson Wongso, Lihuan Li+4

The University of New South Wales · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Human Activity Recognition Wearable Sensing Motion Representation Learning Invariant Representation Learning Human Motion Understanding

Open paper AskPDF arXiv

As wearable and mobile devices become increasingly embedded in daily life, they offer a practical way to continuously sense human motion in the wild. But inertial signals are highly dependent on the sensing setup, including body location, mounting position, sensor orientation, device hardware, and sampling protocol. This setup dependence makes it difficult to learn motion representations that transfer across devices and datasets, and limits the broader use of wearable IMUs beyond closed-set recognition. We introduce AnyMo, a geometry-aware framework for setup-agnostic human motion modeling. AnyMo uses physics-grounded IMU simulation over dense body-surface placements to generate diverse and plausible synthetic signals, pre-trains a graph encoder from paired synthetic placement views and masked partial observations, tokenizes multi-position IMU into full-body motion tokens, and aligns these tokens with an LLM for motion-language understanding. We evaluate AnyMo on three complementary tasks: zero-shot activity recognition across 14 unseen downstream datasets, cross-modal retrieval, and wearable IMU motion captioning, where it improves average Accuracy/F1/R@2 by 11.7%/11.6%/22.6% on HAR, increases zero-shot IMU-to-text and text-to-IMU retrieval MRR by 15.9% and 28.6%, respectively, and improves zero-shot captioning BERT-F1 by 18.8%. These results support AnyMo as a generalist model for wearable motion understanding in the wild. Project page: https://baiyuchen.com/project/AnyMo.

Abstract

Cross-Modal Representation Learning

Explore similar work

AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild

DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation

Modular Retrieval-Augmented Generalization for Human Action Recognition