cs.CVOct 30, 2026
SaveMotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding
Organizations: University of Maryland, College Park · Carnegie Mellon University · Massachusetts Institute of Technology
Abstract
Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
Figures & tables
Figure 1: EMG captures both categorical and continuous structure of hand actions. (Top) For three discrete hand configurations repeated twice— fist , point , and flat —each pose produces a distinct and reproducible EMG signature: fist is marked by a suppressed purple channel ( ) with respect to flat and a pronounced red channel ( ) with respect to point; point exhibits lower overall amplitude, with less purple than flat and less red than fist ; flat is distinguished by pronounced purple and red activation. Sharp grey rectangles reflect the boundaries between hand gestures and rest states. (Bottom) A pick-and-lift sequence combines gradual and abrupt transitions: the hand moves smoothly from rest through pregrasp into grasp , changes sharply as force is applied at lift , then settles—non-instantaneously—into a steady hold . The EMG mirrors this structure, with channels quiet at rest, building subtly through pregrasp and grasp, peaking sharply as the lift begins, and stabilizing during hold.
Figure 2: The bimanual EMG sensing setup of MotorSense . Middle : the MindRove Armband used for data collection, worn on the left and right wrists (top and bottom row, respectively), with colored dots marking the electrode sensors equidistantly arranged around the band. Right : cross-sectional anatomy of the lower forearm with each electrode overlaid. Left : example 8-channel EMG recordings from the left and right wristbands gathered during a manipulation task primarily involving the right hand; each colored trace corresponds to one electrode.
Figure 3: Overview of MotorSense . (A) Sensorimotor Alignment. Windows of bimanual EMG (left) and egocentric video (right) are embedded by an EMG encoder and a video encoder and aligned with a symmetric contrastive objective (Section 3.2 ): the video and EMG embeddings of the same window ( , ) are pulled together, while embeddings from different windows are pushed apart. (B) Downstream Tasks. The learned representations are used for downstream tasks: 1) video diffusion consisting of a single reference frame as input along with EMG control signals, 2) action recognition, 3) 3D hand-object Reconstruction, 4) 3D hand contact prediction.
Figure 4: Self-supervised sensorimotor alignment organizes video and EMG by hand action. (a) t-SNE plot of the joint embedding space for video ( green ) and EMG ( purple ) derived from the two towers of our self-supervised contrastive model (Section 3.2 ). A "modality gap" is observed [ 37 ] between video and EMG clusters. (b) The clips associated with each pair: the video query (left) and the clip whose EMG was retrieved (right). Each retrieved clip performs the query’s hand action—cutting, mixing in a bowl, drying—with a different object, scene, and recording. (c) The EMG embeddings of (a), coloured by verb with marker shape denoting the object. Despite the network never being trained over these labels, windows of the same action form compact regions (e.g. mix salad , scrape bowl ) and verbs group together (the cut family in red). Best viewed zoomed-in and in colour.
| EPIC-Kitchens-100 | SSv2 | Ego-Exo4D | MotorSense | ||||
| Pre-training | Verb | Noun | Action | Action | Keystep | Action | |
| Frozen | None (K400-MAE init.) | 43.85 | 17.74 | 9.93 | 29.17 | 17.72 | 16.86 |
| SimCLR | 43.99 | 24.73 | 13.79 | 24.42 | 27.82 | 20.31 | |
| MoCo | 42.84 | 23.67 | 13.02 | 22.87 | 26.89 | 18.83 | |
| BYOL | 38.50 | 15.64 | 6.58 | 20.21 | 18.05 | 8.68 | |
| Ours | 59.85 | 29.79 | 21.29 | 33.57 | 36.11 | 35.02 | |
Table 1: Action recognition results across benchmarks. Top-1 accuracy (%) with the backbone frozen (top) or fine-tuned end-to-end (bottom) on EPIC-Kitchens-100 [ 10 ] , Something-Something v2 [ 26 ] , Ego-Exo4D [ 27 ] , and an annotated subset of MotorSense . All rows use a Hiera-B backbone initialized from K400-MAE [ 49 ] ; SimCLR, MoCo, and BYOL [ 22 ] are pre-trained on the same MotorSense recordings with RGB-only objectives. Best result per column in bold ; rows give our margin over the best baseline.
| ARCTIC | H2O | HOT3D | ||||
| Pre-training | MPJPE | A-MPJPE | MPJPE | A-MPJPE | MPJPE | A-MPJPE |
| Default | 24.29 † | 38.77 † | 24.28 | 51.20 | 30.85 † | 54.58 † |
| MotorSense (ours) | 21.71 | 35.97 | 19.65 | 44.04 | 28.39 † | 48.85 † |
Table 2: 3D hand reconstruction. Mean per-joint position error (mm; lower is better) relative to the hand root (MPJPE) and in camera coordinates without root alignment (A-MPJPE) on ARCTIC [ 20 ] , H2O [ 34 ] , and HOT3D [ 3 ] , using a hand-only ArcticNet-SF [ 20 ] with a ResNet-50 backbone initialized from ImageNet alone or after sensorimotor alignment on MotorSense . is the change in error from our pre-training.
Figure 5: Qualitative results on ARCTIC [ 20 ] for 3D hand-object reconstruction. Each row shows the RGB input, reconstructions from a vision backbone trained without EMG and with EMG ( MotorSense pre-training), and ground truth. Incorporating EMG during pre-training yields 3D hands and objects that more closely match the ground truth.
| Pre-training | IoU (%) | AUROC (%) |
|---|---|---|
| ImageNet [ 14 ] | 48.41 0.76 | 92.08 0.17 |
| DenseCL [ 61 ] | 48.56 0.91 | 91.27 0.29 |
| PeCLR [ 54 ] | 48.30 0.56 | 91.71 0.52 |
| PixPro [ 65 ] | 50.59 0.66 | 93.36 0.25 |
| Ours | 55.20 0.62 | 97.19 0.15 |
| vs. best baseline |
Table 3: Hand contact reconstruction on MotorSense . IoU (%) between predicted and annotated contact regions on hands in contact, and AUROC (%) for detecting whether a hand is in contact; mean std over three seeds. Best in bold ; is our margin over the best baseline.
| Conditioning | Trainable Params | FVD | LPIPS | PSNR |
|---|---|---|---|---|
| CogVideoX-5B-I2V (supervised) | 5.57B | 772 | 0.35 | 17.3 |
| CogVideoX-5B-I2V (zero-shot) | 0 | 926 | 0.36 | 17.4 |
| language conditioning | 0 | 935 | 0.41 | 18.0 |
| video history | 250M | 373 | 0.30 | 22.0 |
| Ours | 50M | 304 | 0.27 | 22.5 |
Table 4: Image-to-video generation on held-out MotorSense recordings . All methods other than the top row share the same frozen video diffusion backbone and differ only in their conditioning signal and the trainable HyperNetwork/LoRA parameters. Evaluation is performed over a bounding box produced by Aria MPS [ 18 ] (applied jointly to generated and original images). Best results are shown in bold .
Figure 6: The MotorSense data collection workflow. Each session is organized into four stages, flowing left to right: hardware setup (connecting the two MindRove armbands over separate Wi-Fi access points and verifying installation of Aria Studio), device fitting (placing the bands slightly above each wrist, powering on and configuring the Aria glasses, and confirming correct mounting via a clench test), recording (launching the EMG and Aria streams together and proceeding with unscripted manipulation), and post-recording (stopping both streams, offloading Aria recordings via Aria Studio, and resetting the hardware for the next session).
| EPIC-KITCHENS-100 | MotorSense | ||||
| Ablation | Variant | Frozen | Fine-tuned | Frozen | Fine-tuned |
| – | K400 init. | 9.93 | 40.06 | 16.86 | 42.22 |
| – | Ours (reference) | 21.29 | 43.81 | 35.02 | 47.51 |
| Modalities | EMG only | 16.33 | 41.27 | 30.46 | 45.29 |
| IMU only | 15.36 | 41.10 | 25.48 | 43.26 | |
| Electrodes | 4 per arm | 18.88 | 41.86 | 31.57 | 44.62 |
Table 5: Pre-training ablations. Action top-1 accuracy on EPIC-KITCHENS-100 and MotorSense with a Hiera-B backbone; each ablated row changes a single setting of the reference (EMG+IMU, 8 electrodes per arm, same-recording batches, time-resolved embeddings). Single seed per evaluation. Best pre-trained result per column in bold .
Figure 7: Qualitative results for video diffusion . We compare our proposed motor conditioning to a video diffusion model conditioned solely on the video history (unimodal conditioning), for a sequence involving the stirring of tofu in a pan. The video conditioning model misinterprets the action performed, generating a video of a scooping action as opposed to a stirring action, whereas our method correctly generates a video of the stirring of tofu.
Figure 8: Qualitative results for video diffusion . We compare our proposed motor conditioning to a video diffusion model conditioned solely on the video history (unimodal conditioning), for a sequence involving the spreading of mayonnaise with a spoon. The video conditioning model misinterprets the action performed, generating a rotation of the bread as opposed to a spreading action of the spoon, whereas our method correctly generates a video of the spreading of mayonnaise with a spoon.
Figure 9: Qualitative results for video diffusion . We compare our proposed motor conditioning to a video diffusion model conditioned solely on the video history (unimodal conditioning), for a bimanual manipulation sequence involving the peeling of an avocado. The video conditioning model renders the index finger as resting against the thumb, whereas our method correctly positions the index finger against the surface of the avocado for the peeling action.