OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction

Shaqi Luo; Yuanyuan Li; Youhao Hu; Chenhao Yu; Chaoran Xu; Jiachen Zhang; Guocai Yao; Tiejun Huang

cs.ROApr 12, 2026

OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction

Authors: Shaqi Luo, Yuanyuan Li, Youhao Hu, Chenhao Yu, Chaoran Xu, Jiachen Zhang, Guocai Yao, Tiejun Huang, +2 more

Organizations: Beijing Academy of Artificial Intelligence · MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Institute of Technology · Beijing University of Posts and Telecommunications · Peking University

Abstract

UMI-style interfaces enable scalable robot learning, but existing systems remain largely visuomotor, relying primarily on RGB observations and trajectory while providing only limited access to physical interaction signals. This becomes a fundamental limitation in contact-rich manipulation, where success depends on contact dynamics such as tactile interaction, internal grasping force, and external interaction wrench that are difficult to infer from vision alone. We present OmniUMI, a unified framework for physically grounded robot learning via human-aligned multimodal interaction. OmniUMI synchronously captures RGB, depth, trajectory, tactile sensing, internal grasping force, and external interaction wrench within a compact handheld system, while maintaining collection--deployment consistency through a shared embodiment design. To support human-aligned demonstration, OmniUMI enables natural perception and modulation of internal grasping force, external interaction wrench, and tactile interaction through bilateral gripper feedback and the handheld embodiment. Built on this interface, we extend diffusion policy with visual, tactile, and force-related observations, and deploy the learned policy through impedance-based execution for unified regulation of motion and contact behavior. Experiments demonstrate reliable sensing and strong downstream performance on force-sensitive pick-and-place, interactive surface erasing, and tactile-informed selective release. Overall, OmniUMI combines physically grounded multimodal data acquisition with human-aligned interaction, providing a scalable foundation for learning contact-rich manipulation.

Figures & tables

Figure 1: OmniUMI overview. Left: a unified multimodal handheld interface for physically grounded data acquisition, synchronously capturing RGB, depth, trajectory, tactile sensing, internal grasping force, and external interaction wrench, with bilateral gripper feedback and a handheld embodiment for human-aligned demonstration. Middle: multimodal policy learning over visual, tactile, and force-related observations. Right: impedance-compatible deployment, where policy outputs are translated into virtual targets and executed for unified regulation of motion and contact behavior in contact-rich manipulation.

Figure 2: Hardware overview of OmniUMI . The system is organized around a unified handheld embodiment that integrates the custom hub, fisheye and depth sensing, 6-axis force sensing, motion tracking, quick-release mechanism, and tactile-compatible gripper structure. The same gripper-side embodiment is reused across data collection and deployment to improve collection–deployment consistency.

Figure 3: Overview of the multimodal diffusion-policy framework used in OmniUMI . Low-dimensional observations (tactile, wrench, and proprioception) and visual inputs (fisheye RGB and depth) are encoded into an observation condition, which is combined with the diffusion-step embedding to form the global condition for the conditional U-Net. During training, the model predicts denoising trajectories from noisy action chunks, while during inference DDIM sampling produces action chunks that are executed in a receding-horizon manner.

Figure 4: Representative force measurements before and after gravity compensation. Left: raw force signals without gravity compensation, showing strong pose-dependent bias. Right: compensated force signals after applying the calibrated gravity model and mass parameter identification, where the force measurements become significantly more concentrated around zero across different static poses.

Figure 5: Representative internal grasp-force trajectories recorded during the whiteboard eraser pick–wipe–place task under three demonstration settings: direct human demonstration, teleoperation without bilateral force feedback, and OmniUMI with bilateral gripper feedback and a handheld embodiment. OmniUMI produces grasp-force trajectories that are substantially closer to direct human demonstrations in both overall magnitude and temporal modulation, whereas teleoperation without bilateral force feedback leads to less stable and less human-like force regulation.

Figure 6: Representative $F_{z}$ profiles recorded during the wiping phase of the whiteboard eraser pick–wipe–place task under three demonstration settings: direct human demonstration, teleoperation without bilateral force feedback, and OmniUMI with bilateral gripper feedback and a handheld embodiment. Since effective whiteboard cleaning depends critically on stable normal contact during erasing, the $z$ -axis force provides the most informative comparison. OmniUMI produces force trajectories that are much closer to direct human demonstrations, whereas teleoperation exhibits more severe force oscillations during wiping.

Figure 7: Representative tactile interaction trajectories recorded during the whiteboard eraser pick–wipe–place task under three demonstration settings: direct human demonstration, teleoperation without bilateral force feedback, and OmniUMI with bilateral gripper feedback and a handheld embodiment, measured by the $\ell_{2}$ norm of the 126-dimensional marker offset vector over time. OmniUMI produces tactile trajectories that are substantially closer to direct human demonstrations in both amplitude and temporal variation, whereas teleoperation yields noisier and less consistent tactile patterns.

Figure 8: Qualitative results on force-related contact-rich manipulation tasks. Top: force-sensitive manipulation with a full unopened bottle of mineral water. Without grasp-force awareness, the policy fails to lift the bottle; with the proposed force-aware policy, the robot successfully lifts and places the bottle correctly into the tray. Bottom: interactive wrench-based whiteboard erasing. Without wrench-aware control, visible markings remain on the whiteboard; with the proposed wrench-informed policy, the robot successfully wipes the markings off the whiteboard clean.

Figure 9: Qualitative results on tactile-informed selective release. The robot initially grasps two nested transparent cups under minimal gripping force. Top: failure case without sufficient tactile discrimination, where both cups are retained. Bottom: success case with tactile-informed control, where the inner cup is released while the outer cup remains grasped. The accompanying fisheye and tactile observations illustrate the local contact cues used for fine-grained release control.

Method	Success Rate (%) $\uparrow$	Slippage Rate (%) $\downarrow$
w/o Grasping Force	0.0	100.0
w/ Grasping Force (Ours)	100.0	0.0

Table 1: Comparison on the force-sensitive pick-and-place task using a full unopened bottle of mineral water. Higher success rate and lower slippage indicate better internal grasp-force regulation.

Method	Success Rate@5% Residual (%) $\uparrow$	Success Rate@50% Residual (%) $\uparrow$
w/o Wrench	0.0	60.0
w/ Wrench (Ours)	100.0	100.0

Table 2: Comparison on the interactive wrench-based whiteboard erasing task. Two success criteria are reported based on the remaining red-marking area after erasing.

Method	Selective Release Success Rate (%) $\uparrow$	Both Cups Retained Rate (%) $\downarrow$
w/o Tactile	20.0	80.0
w/ Tactile (Ours)	100.0	0.0

Table 3: Comparison on the tactile-informed nested-cup selective-release task. Tactile input improves selective release under minimal gripping force.

Explore similar work

PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
Sep 24, 2026Conor W. Hayes, Rickmer Krohn, Aravind Ramaswami +6Contact-Rich Robotic Manipulation Robotic Data Collection
Center for Robotics and Biosystems, Northwestern Univ., Evanston, IL, USA · Interactive Robot Perception & Learning (PEARL) Lab, TU Darmstadt, Germany · Robotics Institute Germany (RIG)
WT-UMI: Tactile-based Whole-Body Manipulation via Force-Supervised Contact-Aware Planning
Jun 11, 2026Jaehwi Jang, Zhaoyuan Gu, Alfred Cueva +15Contact-Rich Robotic Manipulation Whole-Body Robotic Manipulation
The Institute for Robotics and Intelligent Machines, Georgia Institute of Technology
Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning
Aug 2, 2026Shilin Shan, Chuhao Zhou, Ruize Wang +30Contact-Rich Robotic Manipulation Robotic Tactile Sensing
Nanyang Technological University · Stanford University · University of California, Berkeley +9

Sep 24, 2026J/K move · Enter open · S save

PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation

Conor W. Hayes, Rickmer Krohn, Aravind Ramaswami, Anunth Ramaswami+5

Center for Robotics and Biosystems, Northwestern Univ., Evanston, IL, USA · Interactive Robot Perception & Learning (PEARL) Lab, TU Darmstadt, Germany · Robotics Institute Germany (RIG)

Contact-Rich Robotic Manipulation Robotic Data Collection Robotic Manipulation Robot Imitation Learning Tactile Sensing for Robotic Manipulation Multimodal Sensor Fusion

Open paper AskPDF arXiv

Humans typically rely on vision, touch, hearing, and proprioception to perceive contact and adapt their actions during manipulation. Providing robots with comparable responsiveness therefore requires hardware that can retain and use these complementary sensory signals. Most imitation-learning systems, however, observe demonstrations primarily through vision and proprioception, limiting access to contact information that is difficult to infer visually. We present PolyUMI, an open-source platform for scalable visual--tactile--audio demonstration collection and robot deployment. Its lightweight, wireless handheld gripper records synchronized wrist-camera, optical tactile, contact-audio, and proprioceptive observations without requiring a tethered workstation. The same sensing finger can be transferred to the robot end effector, preserving the sensing geometry between demonstration collection and policy execution. To effectively use these heterogeneous observations, we further introduce VisTA, a token-level multimodal policy that integrates information across sensors and time to predict contact-aware robot actions. Experiments spanning object inference, slip control, and contact-rich manipulation show that touch and audio reveal task-relevant information beyond vision and that VisTA is competitive with or outperforms existing multimodal policies. Together, PolyUMI and VisTA provide an accessible pipeline for collecting multimodal demonstrations and learning policies that perceive physical interaction beyond vision. Project Page: https://polyumi-vista.github.io

Abstract

Explore similar work

PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation

WT-UMI: Tactile-based Whole-Body Manipulation via Force-Supervised Contact-Aware Planning

Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning