Learning Versatile Humanoid Manipulation with Touch Dreaming

Yaru Niu; Zhenlong Fang; Binghong Chen; Shuai Zhou; Revanth Krishna Senthilkumaran; Hao Zhang; Bingqing Chen; Chen Qiu

cs.ROApr 14, 2026

Learning Versatile Humanoid Manipulation with Touch Dreaming

Authors: Yaru Niu, Zhenlong Fang, Binghong Chen, Shuai Zhou, Revanth Krishna Senthilkumaran, Hao Zhang, Bingqing Chen, Chen Qiu, +3 more

Organizations: Carnegie Mellon University · Bosch Center for AI · UT Arlington

Abstract

Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.

Figures & tables

Method	End-Effector Dexterity	Whole- Body	Touch Sensing	Touch Modeling
OmniH2O [ 1 ]	Dex-Hand Full	✓	✗	✗
HumanPlus [ 5 ]	Dex-Hand Full	✓	✗	✗
Mobile-TeleVision [ 20 ]	Dex-Hand Full	✓	✗	✗
AMO [ 21 ]	Dex-Hand Full	✓	✗	✗
ViTacFormer [ 12 ]	Dex-Hand Full	✗	✓	✓
TWIST2 [ 7 ]	Dex-Hand Open/Close	✓	✗	✗

TABLE I : Comparisons to previous humanoid manipulation learning systems

Fig. 2 : System Overview. Left (LBC Training): A teacher-student framework trains the lower-body controller (LBC) to track base velocity, torso orientation, and height, while robustly handling retargeted arm motions from the AMASS dataset. Middle-Left (Teleoperation): Human VR motions are mapped into unified torso commands (for LBC), end-effector poses (for IK), and hand targets (for retargeting), with a joystick dictating base velocity. Middle-Right (Touch Dreaming): A multimodal transformer policy processes vision, touch, and proprioception to predict action chunks alongside future hand joint forces and tactile latents. Future tactile latents are supervised by an EMA target encoder (teacher encoder in Sec. III-E ) with stop-gradient, providing stable latent targets. Right (Deployment): The policy streams action chunks at 30 Hz to the LBC, IK solver, and hand retargeter, all of which operate at 50 Hz.

Fig. 3 : System setup. Hardware used for whole-body humanoid data collection and policy learning, including a dual-lens head camera, wrist cameras, dexterous hands equipped with distributed tactile sensors, and per-joint force feedback from the hand joints. The tactile layout covers the fingers and palms of both hands, and the inset visualizes the corresponding sensor maps together with representative contact activations.

Fig. 4 : HTD model architecture. HTD is a modular encoder–decoder Transformer. Left: modality tokenizers encode multi-view images, proprioception, hand joint forces, and tactile signals into a fixed number of tokens via cross-attention aggregation. Middle: a Transformer encoder fuses multimodal observation tokens, and a Transformer decoder produces a fixed set of output tokens. Right: modular action experts decode pose/velocity/hand-action targets, while modular dream experts predict future forces and tactile latents for touch dreaming. We use learnable query embeddings to flexibly determine how many tokens to use for each input/output modality.

Fig. 5 : Visualization of postures near the boundary of the stable controllable workspace of our LBC policy in simulation.

Metric	Ours	AMO [ 21 ]	FALCON [ 27 ]
$E_{v}$ (m/s)	0.1420 $\pm$ 0.0568	0.1779 $\pm$ 0.0642	0.1641 $\pm$ 0.0309
$E_{\omega}$ (rad/s)	0.1806 $\pm$ 0.0534	0.1540 $\pm$ 0.0316	0.1874 $\pm$ 0.0263
$E_{h}$ (m)	0.0280 $\pm$ 0.0438	0.0568 $\pm$ 0.0814	0.1299 $\pm$ 0.0082
$E_{y}$ (rad)	0.0126 $\pm$ 0.0051	0.1540 $\pm$ 0.0534	0.1215 $\pm$ 0.0111
$E_{p}$ (rad)	0.0487 $\pm$ 0.1796	0.1519 $\pm$ 0.1254	(not tracked)
$E_{r}$ (rad)	0.0157 $\pm$ 0.0065	0.0735 $\pm$ 0.0447	(not tracked)

TABLE II : Tracking Error Comparison

Fig. 6 : Real-world results on five contact-rich tasks. We compare ACT (Visual + Proprio), ACT (Visual + Proprio + Touch), and HTD. Left: success rate. Right: task score (mean

\pm

SEM, 20 trials).

Fig. 7 : Ablations of HTD. Variants: w/o Touch and TD, w/o TD, Dream Raw Tactile, and Dream Latent Tactile (full). Left: success rate. Right: task score (mean

\pm

SEM, 20 trials).

Panel of Figure 9 — Fig. 8 : Touch dreaming visualization. We compare predicted (Pred) versus ground-truth (GT) future contact signals on representative rollouts for two tasks. For each task, the top left shows per-finger hand force trajectories and the mean absolute error (MAE) for the left and right hands, and the bottom left shows the corresponding normalized L2 similarity $s_{\mathrm{L2}}=1-\|\bm{p}-\bm{g}\|_{2}/(\|\bm{p}\|_{2}+\|\bm{g}\|_{2})$ over time, where $\bm{p}$ and $\bm{g}$ denote predicted and ground-truth tactile latents, respectively. The vertical dashed lines in the left plots indicate the specific timestamps for the synchronized camera views and heatmaps of the dreamed versus ground-truth tactile latents shown on the right.

Command	Range	Command	Range
$v_{x}$	$[-0.5,\;0.5]$ m/s	$\phi_{\text{torso}}$ (roll)	$[-0.7,\;0.7]$ rad
$v_{y}$	$[-0.5,\;0.5]$ m/s	$\theta_{\text{torso}}$ (pitch)	$[-0.52,\;1.57]$ rad
$\omega_{z}$	$[-1.57,\;1.57]$ rad/s	$\psi_{\text{torso}}$ (yaw)	$[-1.57,\;1.57]$ rad
$h$	$[0.35,\;0.8]$ m

TABLE III : Command ranges

Parameter	Range	Parameter	Range
Angular velocity	$\pm 0.2$ rad/s	Static friction	$[0.6,\;1.0]$
Projected gravity	$\pm 0.05$	Dynamic friction	$[0.4,\;0.8]$
Joint position	$\pm 0.01$ rad	Restitution	$[0.0,\;0.005]$
Joint velocity	$\pm 1.5$ rad/s	Base mass	$[-5.0,\;5.0]$ kg

TABLE IV : Domain randomizations

Command	Stable range
Torso height $h$	$[0.33,\,0.80]$ m
Torso roll $\phi_{\text{torso}}$	$[-0.38,\,0.35]$ rad
Torso pitch $\theta_{\text{torso}}$	$[-0.92,\,1.41]$ rad
Torso yaw $\psi_{\text{torso}}$	$[-1.50,\,1.34]$ rad

TABLE V : Maximum stable controllable ranges of our LBC policy in simulation.

Term	Weight	Term	Weight
Tracking rewards
Linear velocity $\textstyle r_{\text{vel}}:=\exp\!\left(-\frac{\\|\mathbf{v}_{xy}-\mathbf{v}_{xy}^{\ast}\\|_{2}^{2}}{\sigma_{v}^{2}}\right)$	1.0	Torso roll $\textstyle\begin{aligned} r_{\text{roll}}&:=\exp\!\left(-\frac{(\phi_{p_{y}}-\phi^{\ast})^{2}}{\sigma_{r}^{2}}\right),\\ \widetilde{\mathbf{q}}_{t}&:=\mathbf{q}_{\mathrm{yaw}}(\mathbf{q}_{p})^{-1}\otimes\mathbf{q}_{t},\\ \phi_{p_{y}}&:=\mathrm{wrap}_{[-\pi,\pi]}\!\left(\mathrm{roll}(\widetilde{\mathbf{q}}_{t})\right)\end{aligned}$	1.0
Angular velocity $\textstyle r_{\text{ang}}:=\exp\!\left(-\frac{(\omega_{z}-\omega_{z}^{\ast})^{2}}{\sigma_{\omega}^{2}}\right)$	1.0	Torso pitch $\textstyle\begin{aligned} r_{\text{pitch}}&:=\exp\!\left(-\frac{(\theta_{p_{y}}-\theta^{\ast})^{2}}{\sigma_{p}^{2}}\right),\\ \widetilde{\mathbf{q}}_{t}&:=\mathbf{q}_{\mathrm{yaw}}(\mathbf{q}_{p})^{-1}\otimes\mathbf{q}_{t},\\ \theta_{p_{y}}&:=\mathrm{wrap}_{[-\pi,\pi]}\!\left(\mathrm{pitch}(\widetilde{\mathbf{q}}_{t})\right)\end{aligned}$	1.5
Torso height $\textstyle r_{h}:=\exp\!\left(-\frac{(h-h^{\ast})^{2}}{\sigma_{h}^{2}}\right)$	1.0	Torso yaw $\textstyle\begin{aligned} r_{\text{yaw}}&:=\exp\!\left(-\frac{(\Delta\psi-\psi^{\ast})^{2}}{\sigma_{y}^{2}}\right),\\ \Delta\psi&:=\psi_{\text{torso}}-\psi_{\text{pelvis}}\end{aligned}$	1.0
Regularization
Energy $\textstyle r_{E}:=\left\\|\,\|\bm{\tau}\odot\dot{\mathbf{q}}\|\,\right\\|_{2}$	$-0.001$	Action rate $\textstyle r_{\Delta a}:=\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\\|_{2}^{2}$	$-0.01$

TABLE VI : Reward terms

Explore similar work

Aug 3, 2026

ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction

Fusing tactile signals has proven effective for contact-rich manipulation, enabling robots to perceive contact states and adapt to rapidly changing physical interactions. Yet effectively integrating tactile feedback into dexterous manipulation remains underexplored. In this work, we introduce ReTouch, a vision-language-action model (VLA) that supports contact-rich dexterous manipulation through tactile predictions continually refined online using execution-time feedback. ReTouch builds on two main innovations for tactile representation and closed-loop action generation. First, its Tactile-Patch Encoder represents tactile observations as structured tactile patch features that preserve finger identity and local contact structure, providing contact cues for fine-grained dexterous control. Second, its high-frequency action module jointly predicts future tactile states and action chunks and refines both using incoming tactile feedback during execution. This closed-loop refinement keeps tactile predictions aligned with evolving physical interactions, enabling responsive action correction and improving robustness to contact changes and execution errors. We further introduce XHT-Dataset, comprising 900 real-world demonstrations across seven contact-rich tasks collected on an XHand--UR7e platform, and evaluate ReTouch through closed-loop real-robot experiments. ReTouch surpasses the strongest baseline by 18.4 and 23.8 percentage points in average success rate under standard and challenging conditions, respectively, demonstrating its effectiveness and robustness.

Shiqi Zhang, Xin Zhang, Yedong Shen +9

University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK

Contact-Rich Robotic Manipulation Robotic Manipulation Tactile Sensing for Robotic Manipulation VLMs for Robotics

Ask PDF

Oct 8, 2026

PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io

Xiaomeng Xu, Han Zhang, Zhian Ruan +6

Stanford University · Amazon FAR · Carnegie Mellon University

Whole-Body Robotic Manipulation Contact-Rich Robotic Manipulation Humanoid Robot Control Cross-Embodiment Robot Learning Learning from Demonstration

Ask PDF

Sep 17, 2026

DexTouch-WM: Learning Action-Conditioned Tactile World Models from Human Touch for Dexterous Robot Manipulation

Learning predictive models of contact-rich dexterous manipulation requires dense tactile interaction, but such data are costly to scale on real robots and remain tied to embodiment-specific sensors. We introduce DexTouch-WM, an action-conditioned world model that learns from scalable human touch to jointly predict future RGB observations and bilateral tactile dynamics. Our insight is that human and robot manipulation share transferable contact dynamics when their tactile observations and action spaces are made compatible. We deploy flexible piezoresistive arrays with a shared sensing layout on both human and dexterous robot hands, and retarget human motion into the robot action space so that human interaction can supervise the same dynamics model used for real-robot prediction. DexTouch-WM couples a pretrained video expert with a lightweight tactile expert using anatomy-aware tactile tokens and aligned action conditioning. In human-to-robot scaling experiments, we keep five hours of real-robot supervision fixed while increasing human interaction from 0 to 100 hours, and observe substantial improvements in held-out robot-domain visual, geometric, and contact prediction despite disjoint human and robot task sets. Beyond prediction, we evaluate the world models as surrogate environments for policy evaluation and as generators of synthetic trajectories for real-robot policy learning, showing that scalable human interaction provides a complementary data axis for learning dexterous robot world models.

Yan Qin, Yue Chen, Wenwei Lin +9

HKUST (GZ) · Xspark AI · PKU +1

World Models for Robotics Tactile Sensing Cross-Embodiment Robot Learning Robotic Manipulation Tactile World Models

Ask PDF