Recent humanoid loco-manipulation in articulated environments has largely focused on geometry-dominant interactions, where success is primarily determined by geometric alignment and motion execution. In contrast, forceful multi-finger manipulation requires the robot to adapt to real-world uncertainties in contact and interaction dynamics. We present Tactile-Driven Motor Adaptation (TMA) and an integrated whole-body system for zero-shot sim-to-real forceful humanoid manipulation, and study round-doorknob opening as a challenging benchmark requiring stable multi-finger frictional grasping, sufficient torque generation without slip, force-regulated pushing, and whole-body stability. TMA structures policy learning around interactions observable through tactile feedback, uses cone geometry as an inductive bias for multi-finger contact representation, and adapts motor behavior online from tactile history. An upper-body manipulation policy is trained alongside a pretrained general locomotion controller to coordinate dexterous interaction and whole-body stabilization. Trained entirely in simulation, TMA transfers zero-shot to a Unitree G1 humanoid with BrainCo Revo2 multi-fingered hands and substantially outperforms standard Rapid Motor Adaptation (RMA) and reactive tactile policy baselines across four commercial round doorknobs in the real world.
Figures & tables
Fig. 2: Articulated door mechanism. The door model comprises a spring-loaded knob joint θK , a revolute hinge joint θD , and a translating latch coupled to knob rotation. Knob rotation retracts the latch: once θK reaches the release threshold θrel , the door panel can rotate about the hinge.
Fig. 3: Overview of Tactile-Driven Motor Adaptation. (a) The upper-body manipulation policy commands the right arm and hand, while a pretrained locomotion controller commands the leg and waist joints to maintain whole-body balance under interaction-induced disturbances. The two policies share whole-body proprioceptive feedback, coupling manipulation and balance within an integrated control system. (b) Each fingertip contact force is parameterized in cone coordinates by the normal load n , force cone angle α , and circularly encoded shear direction (sinϕ,cosϕ) . This formulation separates the uncertainty source while preserving the contact mechanics. (c) The policy learning proceeds in two stages. First, the privileged encoder and tactile-centric policy are optimized jointly end-to-end under the task objective, forming an oracle policy and yielding a task-relevant interaction latent. Both modules are then frozen, and the adaptation encoder is trained through latent regression to estimate the interaction latent from a 2 s history of deployable observations. (d) During real-world deployment, the adaptation encoder updates the interaction latent online from tactile interaction history and conditions the frozen tactile-centric policy to generate manipulation motor commands.
Fig. 4: Qualitative real-world rollouts of TMA. Each row presents a commercial knob in front and top views, followed by a successful rollout from left to right: reaching, multi-finger grasping, knob rotation, and door opening. An AprilTag provides knob pose estimation, and tape applied to the knob rim increases contact friction. Timestamps indicate elapsed rollout time.
Fig. 5: Simulation training performance . (a) The privileged policy is pretrained with clean observations and then fine-tuned with observation noise. (b) With the privileged policy frozen, TMA-TConv and TMA-GRU regress its interaction latent over the same 200 -million-sample window.
Method
TMA (Ours)
Binary contact
Normal load
Shear components
Input dims.
16
4
4
8
G (%)
91.3 ± 1.0
88.4±1.2
86.5±1.1
72.8±1.6
S (%)
80.9 ± 1.4
73.6±1.4
74.2±1.5
65.1±1.7
TABLE I: Simulation tactile-signal ablation. Input dimension denotes the per-frame tactile feature size. G and S denote grasp and door-open success rates.
Fig. 6: Quantitative real-world results. Each method is evaluated over 10 trials per knob. Dark bar segments show door-open success S , and light segments show additional grasp-only successes. The total height is the number of grasp successes G . Annotations report S/G .
Fig. 7: Representative real-world baseline failures. Top: the reactive policy, using binary-contact feedback over three observation frames, contacts the knob but fails to open the door. Bottom: RMA, using proprioceptive history without tactile feedback, fails to maintain engagement with the knob. Each sequence progresses from left to right with timestamps.
A policy tuned for one robot often behaves differently on another, whether due to the sim-to-real gap, unknown payloads, or the differing dynamics of two instances of the same robot. In contact-rich, dynamic manipulation, even small motion discrepancies can result in failure to track reference motion, since they disrupt the timing and modes of contact. Common remedies, such as domain randomization or system identification, either produce overly conservative task policies or require data that must be recollected for each robot or payload. We introduce the Torque Adaptation Module (TAM), a learned module that adapts the torque commands sent to the robot to match the behavior of an ideal robot. TAM operates between the low-level controller that tracks the policy's actions and the robot's torque interface. It includes a history encoder that embeds proprioceptive history into a latent state and a torque adaptor that computes residual torque corrections. Because TAM depends only on proprioceptive history and not on policy observations, or the action space, the same TAM weights can be reused to adapt policies with different action spaces (joint targets, end-effector targets, or direct torques). The policies themselves do not need to be trained with domain randomization of robot parameters. Instead, we offload the need for domain randomization to TAM by training it entirely in randomized simulation, using multi-robot pretraining followed by a robot-specific fine-tuning step that still requires no real-robot data. We evaluate TAM zero-shot on a real Franka Panda robot across dynamic manipulation tasks that include a vision-based box pushing policy (from RL), a flip policy (from BC), and an MPC ball-on-plate balancing. Our experiments show that TAM improves zero-shot real-robot execution compared to online system identification and RMA baselines and enables robust dynamic manipulation performance.
Dongwon Son, Florian Shkurti, Jason Lee +3
KAIST · Allen Institute for AI · University of Toronto +1
Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.
Yaru Niu, Zhenlong Fang, Binghong Chen +8
Carnegie Mellon University · Bosch Center for AI · UT Arlington
Contact-rich manipulation requires robots to regulate both motion and interaction forces, yet achieving adaptive compliance remains a fundamental challenge. Learning from real-world data is costly and risky, while simulation-based approaches struggle with the sim-to-real gap in contact dynamics; existing sim-to-real methods either require real-world adaptation or sacrifice adaptive compliance by relying on isotropic compliant controllers. Our key insight is that force regulation decomposes into a time-varying but simulation-transferable directional component and a dynamics-sensitive but manually tunable magnitude component. We instantiate this directional component as two policy outputs, a task frame and a control mode vector, predicted by a visuomotor policy adapted from a pre-trained VLA model and trained via imitation learning on automatically generated simulation demonstrations. At deployment, an admittance controller integrates these predictions with human-specified stiffness and target wrench values to realize adaptive compliance. Our approach achieves adaptive compliance using only simulation data and can benefit from large-scale VLA pre-training. Extensive real-world experiments on four contact-rich tasks, microwave opening, peg-in-hole insertion, whiteboard wiping, and door opening, demonstrate strong task success rates and robustness to external disturbances. Project page: https://yifei-y.github.io/project-pages/TDC/.
Yifei Yang, Anzhe Chen, Zhenjie Zhu +6
Zhejiang University · Zhejiang Humanoid Robot Innovation Center