Learning Versatile Humanoid Manipulation with Touch Dreaming

Yaru Niu; Zhenlong Fang; Binghong Chen; Shuai Zhou; Revanth Krishna Senthilkumaran; Hao Zhang; Bingqing Chen; Chen Qiu

cs.ROApr 14, 2026

Learning Versatile Humanoid Manipulation with Touch Dreaming

Authors: Yaru Niu, Zhenlong Fang, Binghong Chen, Shuai Zhou, Revanth Krishna Senthilkumaran, Hao Zhang, Bingqing Chen, Chen Qiu, +3 more

Organizations: Carnegie Mellon University · Bosch Center for AI · UT Arlington

Abstract

Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.

Figures & tables

Method	End-Effector Dexterity	Whole- Body	Touch Sensing	Touch Modeling
OmniH2O [ 1 ]	Dex-Hand Full	✓	✗	✗
HumanPlus [ 5 ]	Dex-Hand Full	✓	✗	✗
Mobile-TeleVision [ 20 ]	Dex-Hand Full	✓	✗	✗
AMO [ 21 ]	Dex-Hand Full	✓	✗	✗
ViTacFormer [ 12 ]	Dex-Hand Full	✗	✓	✓
TWIST2 [ 7 ]	Dex-Hand Open/Close	✓	✗	✗

TABLE I : Comparisons to previous humanoid manipulation learning systems

Fig. 2 : System Overview. Left (LBC Training): A teacher-student framework trains the lower-body controller (LBC) to track base velocity, torso orientation, and height, while robustly handling retargeted arm motions from the AMASS dataset. Middle-Left (Teleoperation): Human VR motions are mapped into unified torso commands (for LBC), end-effector poses (for IK), and hand targets (for retargeting), with a joystick dictating base velocity. Middle-Right (Touch Dreaming): A multimodal transformer policy processes vision, touch, and proprioception to predict action chunks alongside future hand joint forces and tactile latents. Future tactile latents are supervised by an EMA target encoder (teacher encoder in Sec. III-E ) with stop-gradient, providing stable latent targets. Right (Deployment): The policy streams action chunks at 30 Hz to the LBC, IK solver, and hand retargeter, all of which operate at 50 Hz.

Fig. 3 : System setup. Hardware used for whole-body humanoid data collection and policy learning, including a dual-lens head camera, wrist cameras, dexterous hands equipped with distributed tactile sensors, and per-joint force feedback from the hand joints. The tactile layout covers the fingers and palms of both hands, and the inset visualizes the corresponding sensor maps together with representative contact activations.

Fig. 4 : HTD model architecture. HTD is a modular encoder–decoder Transformer. Left: modality tokenizers encode multi-view images, proprioception, hand joint forces, and tactile signals into a fixed number of tokens via cross-attention aggregation. Middle: a Transformer encoder fuses multimodal observation tokens, and a Transformer decoder produces a fixed set of output tokens. Right: modular action experts decode pose/velocity/hand-action targets, while modular dream experts predict future forces and tactile latents for touch dreaming. We use learnable query embeddings to flexibly determine how many tokens to use for each input/output modality.

Fig. 5 : Visualization of postures near the boundary of the stable controllable workspace of our LBC policy in simulation.

Metric	Ours	AMO [ 21 ]	FALCON [ 27 ]
$E_{v}$ (m/s)	0.1420 $\pm$ 0.0568	0.1779 $\pm$ 0.0642	0.1641 $\pm$ 0.0309
$E_{\omega}$ (rad/s)	0.1806 $\pm$ 0.0534	0.1540 $\pm$ 0.0316	0.1874 $\pm$ 0.0263
$E_{h}$ (m)	0.0280 $\pm$ 0.0438	0.0568 $\pm$ 0.0814	0.1299 $\pm$ 0.0082
$E_{y}$ (rad)	0.0126 $\pm$ 0.0051	0.1540 $\pm$ 0.0534	0.1215 $\pm$ 0.0111
$E_{p}$ (rad)	0.0487 $\pm$ 0.1796	0.1519 $\pm$ 0.1254	(not tracked)
$E_{r}$ (rad)	0.0157 $\pm$ 0.0065	0.0735 $\pm$ 0.0447	(not tracked)

TABLE II : Tracking Error Comparison

Fig. 6 : Real-world results on five contact-rich tasks. We compare ACT (Visual + Proprio), ACT (Visual + Proprio + Touch), and HTD. Left: success rate. Right: task score (mean

\pm

SEM, 20 trials).

Fig. 7 : Ablations of HTD. Variants: w/o Touch and TD, w/o TD, Dream Raw Tactile, and Dream Latent Tactile (full). Left: success rate. Right: task score (mean

\pm

SEM, 20 trials).

Panel of Figure 9 — Fig. 8 : Touch dreaming visualization. We compare predicted (Pred) versus ground-truth (GT) future contact signals on representative rollouts for two tasks. For each task, the top left shows per-finger hand force trajectories and the mean absolute error (MAE) for the left and right hands, and the bottom left shows the corresponding normalized L2 similarity $s_{\mathrm{L2}}=1-\|\bm{p}-\bm{g}\|_{2}/(\|\bm{p}\|_{2}+\|\bm{g}\|_{2})$ over time, where $\bm{p}$ and $\bm{g}$ denote predicted and ground-truth tactile latents, respectively. The vertical dashed lines in the left plots indicate the specific timestamps for the synchronized camera views and heatmaps of the dreamed versus ground-truth tactile latents shown on the right.

Command	Range	Command	Range
$v_{x}$	$[-0.5,\;0.5]$ m/s	$\phi_{\text{torso}}$ (roll)	$[-0.7,\;0.7]$ rad
$v_{y}$	$[-0.5,\;0.5]$ m/s	$\theta_{\text{torso}}$ (pitch)	$[-0.52,\;1.57]$ rad
$\omega_{z}$	$[-1.57,\;1.57]$ rad/s	$\psi_{\text{torso}}$ (yaw)	$[-1.57,\;1.57]$ rad
$h$	$[0.35,\;0.8]$ m

TABLE III : Command ranges

Parameter	Range	Parameter	Range
Angular velocity	$\pm 0.2$ rad/s	Static friction	$[0.6,\;1.0]$
Projected gravity	$\pm 0.05$	Dynamic friction	$[0.4,\;0.8]$
Joint position	$\pm 0.01$ rad	Restitution	$[0.0,\;0.005]$
Joint velocity	$\pm 1.5$ rad/s	Base mass	$[-5.0,\;5.0]$ kg

TABLE IV : Domain randomizations

Command	Stable range
Torso height $h$	$[0.33,\,0.80]$ m
Torso roll $\phi_{\text{torso}}$	$[-0.38,\,0.35]$ rad
Torso pitch $\theta_{\text{torso}}$	$[-0.92,\,1.41]$ rad
Torso yaw $\psi_{\text{torso}}$	$[-1.50,\,1.34]$ rad

TABLE V : Maximum stable controllable ranges of our LBC policy in simulation.

Term	Weight	Term	Weight
Tracking rewards
Linear velocity $\textstyle r_{\text{vel}}:=\exp\!\left(-\frac{\\|\mathbf{v}_{xy}-\mathbf{v}_{xy}^{\ast}\\|_{2}^{2}}{\sigma_{v}^{2}}\right)$	1.0	Torso roll $\textstyle\begin{aligned} r_{\text{roll}}&:=\exp\!\left(-\frac{(\phi_{p_{y}}-\phi^{\ast})^{2}}{\sigma_{r}^{2}}\right),\\ \widetilde{\mathbf{q}}_{t}&:=\mathbf{q}_{\mathrm{yaw}}(\mathbf{q}_{p})^{-1}\otimes\mathbf{q}_{t},\\ \phi_{p_{y}}&:=\mathrm{wrap}_{[-\pi,\pi]}\!\left(\mathrm{roll}(\widetilde{\mathbf{q}}_{t})\right)\end{aligned}$	1.0
Angular velocity $\textstyle r_{\text{ang}}:=\exp\!\left(-\frac{(\omega_{z}-\omega_{z}^{\ast})^{2}}{\sigma_{\omega}^{2}}\right)$	1.0	Torso pitch $\textstyle\begin{aligned} r_{\text{pitch}}&:=\exp\!\left(-\frac{(\theta_{p_{y}}-\theta^{\ast})^{2}}{\sigma_{p}^{2}}\right),\\ \widetilde{\mathbf{q}}_{t}&:=\mathbf{q}_{\mathrm{yaw}}(\mathbf{q}_{p})^{-1}\otimes\mathbf{q}_{t},\\ \theta_{p_{y}}&:=\mathrm{wrap}_{[-\pi,\pi]}\!\left(\mathrm{pitch}(\widetilde{\mathbf{q}}_{t})\right)\end{aligned}$	1.5
Torso height $\textstyle r_{h}:=\exp\!\left(-\frac{(h-h^{\ast})^{2}}{\sigma_{h}^{2}}\right)$	1.0	Torso yaw $\textstyle\begin{aligned} r_{\text{yaw}}&:=\exp\!\left(-\frac{(\Delta\psi-\psi^{\ast})^{2}}{\sigma_{y}^{2}}\right),\\ \Delta\psi&:=\psi_{\text{torso}}-\psi_{\text{pelvis}}\end{aligned}$	1.0
Regularization
Energy $\textstyle r_{E}:=\left\\|\,\|\bm{\tau}\odot\dot{\mathbf{q}}\|\,\right\\|_{2}$	$-0.001$	Action rate $\textstyle r_{\Delta a}:=\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\\|_{2}^{2}$	$-0.01$

TABLE VI : Reward terms

Explore similar work

ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction
Aug 3, 2026Shiqi Zhang, Xin Zhang, Yedong Shen +9Contact-Rich Robotic Manipulation Robotic Manipulation
University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Oct 8, 2026Xiaomeng Xu, Han Zhang, Zhian Ruan +6Whole-Body Robotic Manipulation Contact-Rich Robotic Manipulation
Stanford University · Amazon FAR · Carnegie Mellon University
DexTouch-WM: Learning Action-Conditioned Tactile World Models from Human Touch for Dexterous Robot Manipulation
Sep 17, 2026Yan Qin, Yue Chen, Wenwei Lin +9World Models for Robotics Tactile Sensing
HKUST (GZ) · Xspark AI · PKU +1

Aug 3, 2026J/K move · Enter open · S save

ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction

Shiqi Zhang, Xin Zhang, Yedong Shen, Jiajun Deng+8

University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK

Contact-Rich Robotic Manipulation Robotic Manipulation Tactile Sensing for Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Fusing tactile signals has proven effective for contact-rich manipulation, enabling robots to perceive contact states and adapt to rapidly changing physical interactions. Yet effectively integrating tactile feedback into dexterous manipulation remains underexplored. In this work, we introduce ReTouch, a vision-language-action model (VLA) that supports contact-rich dexterous manipulation through tactile predictions continually refined online using execution-time feedback. ReTouch builds on two main innovations for tactile representation and closed-loop action generation. First, its Tactile-Patch Encoder represents tactile observations as structured tactile patch features that preserve finger identity and local contact structure, providing contact cues for fine-grained dexterous control. Second, its high-frequency action module jointly predicts future tactile states and action chunks and refines both using incoming tactile feedback during execution. This closed-loop refinement keeps tactile predictions aligned with evolving physical interactions, enabling responsive action correction and improving robustness to contact changes and execution errors. We further introduce XHT-Dataset, comprising 900 real-world demonstrations across seven contact-rich tasks collected on an XHand--UR7e platform, and evaluate ReTouch through closed-loop real-robot experiments. ReTouch surpasses the strongest baseline by 18.4 and 23.8 percentage points in average success rate under standard and challenging conditions, respectively, demonstrating its effectiveness and robustness.

Abstract

Explore similar work

ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction

PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

DexTouch-WM: Learning Action-Conditioned Tactile World Models from Human Touch for Dexterous Robot Manipulation