CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video

Zhuoqun Chen; Shucheng Jia; Boyuan Chen

cs.ROOct 4, 2026

CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video

Authors: Zhuoqun Chen, Shucheng Jia, Boyuan Chen

Organizations: Duke University

Abstract

Partnered human-humanoid interaction couples locomotion with continuous physical contact. A humanoid needs to coordinate with a person's motion while responding to interaction forces and maintaining stable and natural movement. We present CoDance, a framework for learning reactive and compliant human-humanoid interaction from video. We study partnered dancing as a challenging instantiation, where a humanoid coordinates its footsteps with a moving partner and maintains continuous two-hand contact. Given a single video of two human dancers, CoDance retargets their motions into a robot reference and a moving partner. We introduce a multi-link compliance augmentation that transforms the kinematic demonstration into force-aware training data by adapting the robot reference under structured forces at both hands. Policies trained on this data follow the observed partner while preserving the demonstrated locomotion style and responding compliantly to physical interaction. In simulation, the policies adapt their footsteps to changes in the partner and reproduce approximately 80% of the wrist displacement encoded by the augmented demonstrations. On a physical humanoid, CoDance enables sustained two-hand dancing with a human partner including repeated transitions between forward and backward motions.

Figures & tables

Fig. 2 : Overview of CoDance . A paired-dancing video is converted into force-aware training data through motion retargeting and multi-link compliance augmentation. The policy learns to follow the observed partner, maintain the demonstrated locomotion style, and remain compliant at both hands. At deployment, the simulated partner is replaced by a person. Training is illustrated with the decoupled policy.

Parameter	Value
Robot stiffness $K$	log-uniform, $101000$ $\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$
Environment stiffness	same range, independent draw
Force ceiling	$140\text{\,}\mathrm{N}$
Displacement scale $d_{\max}$	$0.15\text{\,}\mathrm{m}$
per-event draw	$\mathcal{U}(0.03,d_{\max})$ m
Balance budget (CoM shift)	$0.10\text{\,}\mathrm{m}$

TABLE I : Augmentation parameters.

Fig. 3 : Evaluation results on compliance interaction. Under force, both compliant policies move the wrist most of the distance encoded in the adapted reference, while the stiff tracker moves it far less. Error bars show the standard deviation over evaluation seeds.

Fig. 4 : Visualization of the free and adapted references (top) and of the decoupled policy with and without the adversarial term, at the middle of four force holds. The partner is in teal, and arrows mark the force at the left wrist (blue) and right wrist (pink), with the applied force solid and the scheduled force faded. Without the adversarial term, the pelvis turns away from the dancer’s heading, the waist and hips twist back toward the direction of travel, and the stance widens into a crouch. Top row also shows that separate retargeting does not preserve relative position in the paired motion clip.

Fig. 5 : Visualization of one real-robot dance with a human partner. The robot keeps close to the following distance for most of the dance. Top: the trajectories of the robot and the partner. Bottom: the partner’s root in the robot’s pelvis frame.

Fig. 6 : Visualization of a standing policy under force on one or both hands. The red overlay is the reference standing pose. Left: one hand loaded by a hanging weight. Middle: the two hands pulled in different directions. Right: both hands pulled outward. The arms move with the pull and the feet stay still.

Term	Decoupled	Whole-body	Parameters
Partner following
Right foot follows the partner’s left foot	2	2	offset $0.5\text{\,}\mathrm{m}$ behind, $\sigma$ = $0.25\text{\,}\mathrm{m}$ , deadband $0.05\text{\,}\mathrm{m}$ , height tracked
Left foot follows the partner’s right foot	2	2	same
Feet air time	0.5	0.5	air time between $0.2\text{\,}\mathrm{s}$ and $0.6\text{\,}\mathrm{s}$
Upper-body compliance
Arm positions, anchor-relative	1	1	$\sigma$ = $0.3\text{\,}\mathrm{m}$ ; shoulder, elbow, wrist links

TABLE V : Reward terms of the two trained policies. Tracking terms are exponential kernels

\exp(-e^{2}/\sigma^{2})

of the error

e

with the listed

\sigma

; weights multiply the per-step term. The two policies share every term except the locomotion objective: the decoupled policy draws its locomotion style from the adversarial term, the whole-body tracker (in the style of SoftMimic) from reference tracking of the lower body of the adapted reference.

Term	Per step	History	Actor noise	Actor	Critic
Base linear velocity (IMU)	3	5			✓
Base angular velocity (IMU)	3	5	$\pm 0.2$	✓	✓
Joint positions	29	5	$\pm 0.01$	✓	✓
Joint velocities	29	5	$\pm 0.5$	✓	✓
Previous actions	29	5		✓	✓
Projected gravity	3	5	$\pm 0.05$	✓	✓

TABLE VI : Observations of the two trained policies, identical for both. Per-step sizes are stacked over the listed history. Noise is uniform in the listed range and applied to the actor only; the actor’s joint positions also carry the per-joint encoder bias of Table VII . Both networks are MLPs with hidden layers 512, 256, 128 and ELU activations. The AMP discriminator of the decoupled policy reads 2 frames of 63 features (positions and 6D orientations of 7 lower-body and torso links in the pelvis frame), which the policy itself does not see.

Setting	Value
Randomization (at episode start)
Torso centre of mass offset	uniform, $x$ $\pm$ $0.025\text{\,}\mathrm{m}$ , $y$ $\pm$ $0.05\text{\,}\mathrm{m}$ , $z$ $\pm$ $0.05\text{\,}\mathrm{m}$
Foot friction coefficient	uniform, $0.31.2$ , one draw for all foot geoms
Joint encoder bias	uniform, $\pm$ $0.01\text{\,}\mathrm{rad}$ per joint, added to the actor’s joint positions
Actor observation noise	uniform, per observation term, at the ranges the policy trained with
Reset root pose	$x$ , $y$ $\pm$ $0.15\text{\,}\mathrm{m}$ , yaw $\pm$ $0.3\text{\,}\mathrm{rad}$ about the clip pose

TABLE VII : Randomization, force replay and episode settings, identical for both policies.

Explore similar work

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
Sep 15, 2026Liu Cao, Xingze Wu, Jingzhi Cui +4Contact-Rich Dexterous Manipulation Humanoid Loco-Manipulation
1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong +1
Robotizing Human Videos with Physically Consistent Interactions
Oct 5, 2026Ching-Lam Cheng, Shengfeng He, Bin ZhuHand-Object Interaction Detection
Singapore Management University
HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Sep 28, 2026Jihwan Shin, Adrià López Escoriza, Junzhe He +2Cross-Embodiment Retargeting Articulated Objects
Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Sep 15, 2026, cs.ROJ/K move · Enter open · S save

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu+3

1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong+1

Contact-Rich Dexterous Manipulation Humanoid Loco-Manipulation Articulated Objects Dexterous Manipulation Whole-Body Weave

Open paper AskPDF arXiv

Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/

Abstract

Explore similar work

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

Robotizing Human Videos with Physically Consistent Interactions

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction